https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15252 Lustre <p>When LSOM is not used at cluster it is better to disable it, I don't see such option for now.<br/> During analyze of MDS vmocre with high load avarage, we have found LSOM feature add big impact to it.<br/> 205 threads were blocked with <tt>mdt_lsom_update()</tt>. A few threads got further and were waiting for the osd lock for read. It seems that <tt>mdt_lsom_update()</tt> has a serious issue with a single shared file because of its mdt-level mutex for every close request.</p> LU-15252

option to disable LSOM updates

Improvement Major Resolved Fixed Alexander Boyko Alexander Boyko patch Fri, 19 Nov 2021 09:21:03 +0000 Fri, 26 May 2023 11:16:20 +0000 Thu, 23 Dec 2021 14:23:02 +0000 Lustre 2.15.0 Lustre 2.15.0 0 7 <p>"Alexander Boyko &lt;alexander.boyko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45619" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45619</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15252" title="option to disable LSOM updates" class="issue-link" data-issue-key="LU-15252"><del>LU-15252</del></a> mdc: add client tunable to disable LSOM update<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e10a14aa558f7043ef91f6aae6326d6694c1cacc</p> <p>It would be better to fix the reason why LSOM updates are slow. From the comments, this is due to lock contention on the MDS, but there should be a way to avoid it, since this is "lazy" since and does not have to be totally accurate all the time. Fro example, checking if the incoming LSOM size/blocks is already smaller than current size/blocks without the lock, since LSOM should only be increasing. </p> <p>Also, it may be possible to batch LSOM updates in memory for a few seconds as long as the open counter &gt; 0, since we know/expect some later close will write it to disk, and the update does not need to be part of a transaction if there is no other reason for it. </p> <p>Similarly, it should be possible to cache flag on the mdt_object if there is no LOV EA or DoM is ised, since this changes very rarely, so reading the LOV EA just for these two bits of information is expensive. That would allow checking whether an LSOM update is needed without the object mutex. </p> <p> I agree with Andreas. LSOM is too critical to disable for us and we don't want our MDS servers bogged down at the same time.</p> <p>Andreas, I've made a quick fix for Lustre clients only. But,I agree with you, LSOM requires fixes on the server side to improve single shared file perfomance. By default LSOM is enabled still, so it has no any impact by default.</p> <p>&gt;Similarly, it should be possible to cache flag on the mdt_object if there is no LOV EA or DoM is ised, since this changes very rarely, so reading the LOV EA just for these two bits of information is expensive. That would allow checking whether an LSOM update is needed without the object mutex.<br/> Maybe it is better to store LSOM at mdt object and update xattr only for a last close? With this case only failover would affect lazy size to be wrong at xattr, but I think it is normal for LSOM.<br/> Andreas, any objection?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aboyko" class="user-hover" rel="aboyko">aboyko</a>, I think the LSOM update can be <em>fairly</em> lazy, and there isn't a serious danger if some updates are lost, but there should still be occasional writes of new LSOM data to disk:</p> <ul class="alternate" type="square"> <li>if the inode is being written already for some other reason (e.g. atime update, link count, etc.)</li> <li>after some long time since the last LSOM update (e.g. 60s, like <tt>atime_diff</tt>). maybe LSOM and atime writes will happen at the same time?</li> <li>when the last client closes the file, including when the client is evicted</li> </ul> <p>My main concern would be that files which are <em>never</em> properly closed will also never get LSOM updates. That could happen with config files or shared libraries for jobs that run a very long time, and/or files that are being accessed by different jobs and always have <em>some</em> client process holding them open.</p> <p>"Alexander Boyko &lt;alexander.boyko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45709</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15252" title="option to disable LSOM updates" class="issue-link" data-issue-key="LU-15252"><del>LU-15252</del></a> mdt: reduce contention at mdt_lsom_update<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 495a7eb370cf9dbb5eec67bbd0a59ae206cdb68a</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45619/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45619/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15252" title="option to disable LSOM updates" class="issue-link" data-issue-key="LU-15252"><del>LU-15252</del></a> mdc: add client tunable to disable LSOM update<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 19172ed37851fdd5731b1319c12151f5cb1fe267</p> <p>Landed for 2.15</p> <p>Still a second patch in flight that fixes the performance issue instead of working around it. </p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45709/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45709/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15252" title="option to disable LSOM updates" class="issue-link" data-issue-key="LU-15252"><del>LU-15252</del></a> mdt: reduce contention at mdt_lsom_update<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c8b7afe4970415f8dae84f5e20661f8a3b3681a0</p> Duplicate Development Rank 1|i02ahb: Rank (Obsolete) 9223372036854775807