https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12368 Lustre <p>If multiple threads on a client are executing <tt>statfs()</tt> calls concurrently, and the <tt>obd_statfs()</tt> cache has expired, then each thread will send an <tt>OST_STATFS</tt> RPC to each OST. With certain <tt>statfs</tt>-heavy workloads on many-core client nodes, this can result in thousands of needless RPCs being sent from each client every few seconds.</p> <p>Since all of the callers funnel through <tt>obd_statfs()</tt>, and there is no benefit to having multiple <tt>OST_STATFS</tt> or <tt>MDS_STATFS</tt> replies from the same target (they return the same data, and all threads are blocked on the reply) it makes sense to just allow one thread to execute the statfs and other threads to (interruptibly) wait for it to complete.</p> LU-12368

concurrent statfs() calls on the client should be blocked

Improvement Minor Resolved Fixed Andreas Dilger Andreas Dilger easy Fri, 31 May 2019 19:22:04 +0000 Wed, 12 Aug 2020 17:55:27 +0000 Fri, 8 Nov 2019 18:13:58 +0000 Lustre 2.10.8 Lustre 2.12.3 Lustre 2.13.0 0 7 <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35311" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35311</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> ptlrpc: make DEBUG_REQ messages consistent<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5548152268223786e8acc9df6d470672d8b6e403</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35380" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35380</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> obdclass: don't send multiple statfs RPCs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: acb1b155fd18a7bc835c9d31ffb7fa06e2c009fa</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35383</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> obdclass: allow 'lfs df' to specify cache age<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2448d63aed511658f4b1b6e61789271c77fb0390</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35380/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35380/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> obdclass: don't send multiple statfs RPCs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1c41a6ac390bf74a135861efcd576a3b433d3c49</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35485" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35485</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> obdclass: don't send multiple statfs RPCs<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f0819a6e0b676fb80b9a612ce5c48786242da864</p> <p><a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> testing is underway right now at the customer site, running ExaScaler 4.2 and DDN Lustre 2.10.7_ddn5. We noted the default llite.*.statfs_max_age was set to 1. (or 1 second)</p> <p>When testing with the default age of 1, we observed a regular occurrence (~1 in 10) of slow app timesteps in which each timestep made numerous statfs calls.</p> <p>Increasing the statfs_max_age to 30 seconds, we did see a marked, positive change in behavior in which we only saw 1 in 900 slow app timesteps.</p> <p>Bob, the default statfs cache age has been 1s since the beginning of Lustre. Increasing it to 30s is pretty reasonable, as (IMHO) applications shouldn't really put too much stock in the amount of free space in the filesystem anyway, given that there may be a large number of jobs running concurrently and allocating and freeing space, space is reserved by clients, quota limits may intervene before the filesystem runs out of space, etc. Either there is lots of free space, and the application shouldn't care, or there <em>isn't</em> lots of space and the application may run out of space <em>even</em> if it checks in advance (e.g. some other process consumes the remaining space).</p> <p>Is there any clear indication <em>why</em> the application is doing so many <tt>statfs()</tt> calls?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35485/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35485/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> obdclass: don't send multiple statfs RPCs<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 1839debbba0e70dd5cc9f11c8bc83bcead0114d4</p> <p>Andreas,</p> <p>Any reason this ticket is still open?  I think we've got the patches all merged.</p> <p>Patrick, there is still patch <a href="https://review.whamcloud.com/35383" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35383</a> "<tt>obdclass: allow 'lfs df' to specify cache age</tt>" still open. I added that patch while working on "<tt>obdclass: don't send multiple statfs RPCs</tt>" but I'm not sure if it is actually useful to land or not.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35311/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35311/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> ptlrpc: make DEBUG_REQ messages consistent<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c0fa0ba4a8efcd774f1fe27986a0217c76dedf6d</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36874" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36874</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12368" title="concurrent statfs() calls on the client should be blocked" class="issue-link" data-issue-key="LU-12368"><del>LU-12368</del></a> ptlrpc: make DEBUG_REQ messages consistent<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: bcf0777c27333aade617e92b9cfe03877e568e4e</p> <p>In fact - fix is incorrect. Lets look to <a href="https://jira.whamcloud.com/browse/LU-13296" title="statfs isn&#39;t work properly with MDT statfs proxy" class="issue-link" data-issue-key="LU-13296"><del>LU-13296</del></a>.</p> Related LU-13296 Development Rank 1|i00hc7: Rank (Obsolete) 9223372036854775807