https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15821 Lustre <p>The current code places bl_ast lock callbacks at the end of the global BL callback queue.  This is bad because it causes urgent requests from the server to wait behind non-urgent cleanup tasks to keep lru_size at the right level.</p> <p>This can lead to evictions if there is a large queue of items in the global queue so the callback is not serviced in a timely manner.</p> <p>Put bl_ast callbacks on the priority queue so they do not wait behind the background traffic.</p> LU-15821

Server driven blocking callbacks can wait behind general lru_size management

Bug Minor Resolved Fixed Patrick Farrell Patrick Farrell Thu, 5 May 2022 00:55:11 +0000 Wed, 2 Aug 2023 12:49:40 +0000 Wed, 3 Aug 2022 13:25:57 +0000 Lustre 2.16.0 Lustre 2.15.4 0 7 <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47215" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47215</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 65a5b8d27e6d6a0acf8bc87458b8837509e60b23</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47215/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47215/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2d59294d52b696125acc464e5910c893d9aef237</p> <p>Landed for 2.16</p> <p><del>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48122" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48122</a></del><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Fix unsafe blwi access</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 97326667eb8f960fa2996099a6bd2b96496d026e</del><br/> (Patch not needed)</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48764" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48764</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 74666c9fff24126922e5635fbaf2394bf7eda118</p> <p>It would be nice to have this patch backported to 2.15.x, we have been running it for a while on 2.15.1 clients with good results.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49610</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: b3e9e0eeadd783d22065871df28ea32f2d3c6934</p> <blockquote> <p> we have been running it for a while on 2.15.1 clients with good results.</p></blockquote> <p>Stephane, by "good results" do you mean "it doesn't cause problems" or "it visibly improved/removed some problem that you were seeing with client evictions"? In the use case that drove the initial development of this patch it didn't totally solve the issue. Yang Sheng also just developed patch <a href="https://review.whamcloud.com/49527" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49527</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16285" title="Prolong the lock BL timeout" class="issue-link" data-issue-key="LU-16285"><del>LU-16285</del></a> ldlm: improvement of bl lock queue</tt>" to further improve the handling of highly-contended DLM locks.</p> <p>Andreas, since we have applied this patch, last October 2022, we have not seen again the following problems from two workloads that were previously causing trouble:</p> <ul> <li>GNU parallel with --tmpdir</li> <li>sort with --temporary-directory</li> </ul> <p>In both use cases, files are created in temporary directory and are used unlinked, not visible in the directory but actually still open, this may have triggered some sort of contention in Lustre leading to evictions.</p> <p>But at the same time, we have also tried to redirect our users to local scratch filesystems to avoid further issues, as a parallel filesystem was not really needed. So I can't tell you for sure that this patch resolves these issues, but at least it didn't introduce anything bad and we would like to keep it for now. It would be convenient for us if it was added to 2.15, but otherwise, I will just continue to backport it. I hope that the context helps a bit.</p> <p>Thanks also for the pointer to the other patch from Yang Sheng.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49610/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49610/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15821" title="Server driven blocking callbacks can wait behind general lru_size management" class="issue-link" data-issue-key="LU-15821"><del>LU-15821</del></a> ldlm: Prioritize blocking callbacks<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 8ca1186151faa778edd5abd361e92fcd5d8ff56b</p> Related LU-16285 LU-15822 LU-15915 Development Rank 1|i02p1j: Rank (Obsolete) 9223372036854775807 Severity