https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3308 Lustre <p>Shared directory unlinks using metabench seem significantly slower with Lustre 2.x clients versus Lustre 1.8.x</p> <p>512 processes, 8 ppn, 2.1 server<br/> 1.8.6 clients: 18k unlinks/s<br/> 2.3 clients: 7k unlinks/s</p> <p>creates are comprable at 28k creates/s</p> <p>Mdtest shows no such regression. Digging into metabench a little, it seems that when deleting files, metabench processes (I am told) readdir, select "the next" file, and delete it, effectively racing each other, whereas mdtest deterministically choses the files to delete based on process id.</p> <p>This seems to imply it's directory locking contention on the MDT.</p> <p>On the clients, the majority difference in time is spend in ptlrpc_queue_wait (not sure of units):<br/> 1.8.8: 347<br/> 2.3: 923</p> <p>On the MDT, the big difference is mdt_object_find_lock:<br/> 1.8.8: 51us<br/> 2.3: 1110us</p> <p>Also, using ldlm stats, it seems the 2.3 clients cause twice as many ldlm_bl_callbacks as 1.8.8 clients.</p> <p>So apparently the 2.3 client is holding directory locks differently than the 1.8 clients. We're still looking into this, but if anyone has thoughts we'd love to hear them. </p> LU-3308

large readdir chunk size slows unlink/"rm -r" performance

Bug Major Reopened Unresolved WC Triage Nathan Rutman lug23dd medium performance Fri, 10 May 2013 00:03:39 +0000 Thu, 1 Feb 2024 00:31:24 +0000 Lustre 2.1.6 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.12.0 0 19 <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/LELUS-138" class="external-link" target="_blank" rel="nofollow noopener">LELUS-138</a></p> <p>Is this the metabench you are referring to? <a href="http://www.7byte.com/?page=metabench" class="external-link" target="_blank" rel="nofollow noopener">http://www.7byte.com/?page=metabench</a></p> <p>Have you hand a chance to spin Master lately?</p> <p>Keith, the Metabench code we're using is attached to <a href="https://jira.whamcloud.com/browse/LU-1167" title="Poor mdtest unlink performance with multiple processes per node" class="issue-link" data-issue-key="LU-1167">LU-1167</a>. It is not the version from 7byte.com. Yes, master (well 2.3.63) isn't any better at all. Maybe even a little worse <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>It would be useful to get the performance of 2.1 clients as well, to see how it stacks up.</p> <p>The client ptlrpc_queue_wait() is just a side-effect of the client waiting for the server to complete the requests. It might be useful to gather stats (strace?) to see which operations it is waiting on - unlink() or readdir()</p> <p>One area to check is that the 2.3+ clients will do readdir in large chunks compared to earlier versions. If metabench clients are continually doing readdir and unlinking files, then this might cause a larger amount of data to be sent to all of the clients. An application might do the readdir() once at the beginning, then delete entries <tt>(N % num_clients + client_no)</tt> so that it only has to do readdir once. Looking at the metabench code, it depends on how large the cache in gibc is, since it is calling <tt>readdir()</tt> for every entry. It also appears to be doing <tt>lstat(filename)</tt> on every inode before deleting it, definitely a case of lock ping-pong.</p> <p>I wonder if ELC broke and that leads to bigger number of blocking callbacks in 2.3?<br/> Do you guys have some traces or what sort of analysis did you perform already?</p> <p>Yes we do have traces and analysis to share, which will be coming shortly.</p> <p>I don't have 2.1 client performance handy, but I can tell you that 2.2 gives similar results. In fact, 2.2, 2.3, and master/2.4 (2.3.63) are all in the same neighborhood. Interestingly enough, removing the lstat() completely didn't change the rates at all. I don't know if that necessarily throws out any theories.</p> <p>Also, Nathan noted that single dir unlink scenario in mdtest shows no such regression. But it is also worth noting that the same Metabench behavior of readdir()+lstat()+unlink() doesn't show any regression for the multi-dir case. Would that kind of information rule out the theory on any ELC breakage?</p> <p>We have verified that removing the readdir() + lstat() from Metabench creates mdtest-like results. Using a 2.2 server (with pdirops) did not correct the regression. <span class="error">&#91;It may have boosted the performance, but we didn&#39;t gather a baseline on the test HW/SW config&#93;</span>.</p> <p>Performance of the readdir&amp;lstat-less Metabench version on 2.3 was on-par to even the Metabench multidir case (server @ 2.2). Removing the readdir()+lstat() from Metabench had little effect on 1.8.x client runs.</p> <p>Previous tests with only lstat removed had little to no effect on 2.x clients, but we plan to quickly introduce it to the readdir-less version to further validate that the regression is caused by the interference of readdir with unlink.</p> <p>can statahead be involved somehow? I'd suggest to look at rpc stats on all the sides.</p> <p>Cory:<br/> Can you please disable stat-ahead and then run two metabench tests: one like the original with everything in place and one with just lstat() removed?<br/> I suspect that in this configuration removing lstat will be helpful and further highlight ELC breakage of some sort.</p> <p>Xyratex have tested patch with tunable readdir number of pages for data transfer. Here is the result of running 8 clients, 8ppn and total 64k files:</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>&nbsp;</th> <td class='confluenceTd'>create</td> <td class='confluenceTd'>unlink</td> </tr> <tr> <th class='confluenceTh'>2.3 client 1-page readdir chunk size</th> <td class='confluenceTd'>14380</td> <td class='confluenceTd'>8938</td> </tr> <tr> <th class='confluenceTh'>2.3 client 2-page readdir chunk size</th> <td class='confluenceTd'>14339</td> <td class='confluenceTd'>8419</td> </tr> <tr> <th class='confluenceTh'>2.3 client 32-page readdir chunk size</th> <td class='confluenceTd'>14296</td> <td class='confluenceTd'>6176</td> </tr> <tr> <th class='confluenceTh'>2.3 client 256-page readdir chunk size (default)</th> <td class='confluenceTd'>14247</td> <td class='confluenceTd'> 3739</td> </tr> <tr> <th class='confluenceTh'>1.8.8 client</th> <td class='confluenceTd'>15026</td> <td class='confluenceTd'>9819</td> </tr> </tbody></table> </div> <p>Big readdir chunk size in 2.x client does cause the lock ping-pong and impact the overall unlink performance for shared directory.</p> <p>Alexander, thanks for the data.</p> <p>In this case it makes sense for the client or MDS to auto-tune the readdir chunk size based on usage. If there are readahead directory pages that are being dropped from cache without being accessed, then it makes sense to reduce the readdir chunk size automatically. Unfortunately, I don't think there is any way for the kernel to know that "rm -r" or metabench is the one calling readdir() the first time, so it would still make sense to read a full 1MB of readdir data on first access, and only drop the readdir() chunk size on later access.</p> <p>Ok, so I think the correct approach here is not to reduce the transfer size, but instead we need to use the other idea we dismissed in the past:</p> <p>POSIX allows us to retain cached pages for the readdir purposes as long as the filehandle remains open, so if we do that now, we'll still have all the benefits of fast rm (even faster due to no need to refresh the pages all the time) and at the same time avoid needless page pingpong for readdir.</p> <p>In the past this was deemed unimportant since we did not do statahead so extra pingpong was pretty minimal, but now it becomes more important I think.</p> <p>@Oleg - but don't the cached readdir pages have to be dropped if other clients are simultaneously deleting files? Are you saying POSIX allows for operation on stale directory entries?</p> <p>What I mean is posix allows for pages to remain cached for the application that has the filehandle open, for as long as the filehandle remains open.<br/> All other users should get their own pages that are fresh at the time of getting.</p> <p>So basically now when lockss get cancelled, we are just dropping pages from cache completely.<br/> What we'll need to do instead is to remove reference of them from generic mapping (truncate), but don't drop them completey, instead reference them from all the opened filehandles. Then when doing readding we'll check if we have such pages referenced from the filehandle before doing a lookup in the main cache, and if both are a no, then do an RPC.<br/> Pages would get dereferenced on close, so when the last file opener goes away, they'd be freed (obviously will need some other form of truncation in case of tight memory too).</p> <p>The relevant sections in the readdir(2) definition in SUSv2 <a href="http://pubs.opengroup.org/onlinepubs/007908799/xsh/readdir.html:" class="external-link" target="_blank" rel="nofollow noopener">http://pubs.opengroup.org/onlinepubs/007908799/xsh/readdir.html:</a></p> <blockquote> <p>... Directory entries represent files; files may be removed from a directory or added to a directory asynchronously to the operation of readdir().</p> <p>If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified.</p> <p>The readdir() function may buffer several directory entries per actual read operation;</p></blockquote> <p>So, the readdir() processing does NOT need to be kept coherent w.r.t. entries being added or removed to the directory until rewinddir() is called on that stream.</p> <p>If this is still an area of interest for someone to fix, to start with the <tt>mdc_obd_max_pages_per_rpc</tt> lproc entry needs to be fixed to allow it to be written (shrunk at least) so that <tt>lctl set_param mdc.*.max_pages_per_rpc=N</tt> is allowed. See comment in <tt>mdc/lproc_mdc.c</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * FIXME: below proc entry is provided, but not in used, instead * sbi-&gt;sb_md_brw_size is used, the per obd variable should be used * when CMD is enabled, and dir pages are managed in MDC layer. * Remember to enable proc write function. */ { .name = <span class="code-quote">"max_pages_per_rpc"</span>, .fops = &amp;mdc_obd_max_pages_per_rpc_fops }, </pre> </div></div> <p>At that point it would be easy to do a few benchmarks on a decent sized filesystem (a few million files with a variety of directory sizes, including some larger dirs @ 10K+ entries) to see what effect the readdir BRW size has on real-world readdir performance. Clearly there will be some advantage for readdir of larger directories, but if this is outweighed by the performance loss for unlinks in a shared directory then it might be possible to have a reduced readdir size by default (e.g. 32KB or 64KB) that still gives decent readdir performance but reduces overhead for unlinks.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26088" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26088</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a> mdc: allow setting readdir RPC size parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dd8181ad51af55f7317e7087adb1cad33338158f</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26212" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26212</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a> tests: fix sanity/sanityn test_mkdir() usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6c5270804141c1c4b48be8d357a4686f42c70e18</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26088/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26088/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a> mdc: allow setting readdir RPC size parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 664bad91b5c0e2a5c9423d4b524d9906db9ef9b5</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26212/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26212/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a> tests: fix sanity/sanityn test_mkdir() usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c75aa6c74cd86c69de893395735a6725571f11f4</p> <p>Landed for 2.11</p> <p>This is not actually fixed yet, just minor cleanup patches landed. </p> <p>with c75aa6c74cd86c69de893395735a6725571f11f4 landed I'm getting the following in the local testing:</p> <p>== sanity test 102d: tar restore stripe info from tarfile,not keep osts == 08:27:52 (1506058072)<br/> mkdir: cannot create directory `/mnt/lustre/d102d.sanity': File exists<br/> sanity test_102d: @@@@@@ FAIL: mkdir '/mnt/lustre/d102d.sanity' failed </p> <p>setup_test102() creates d102d.sanity, then test_102d() tries to create it too</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=nrutman" class="user-hover" rel="nrutman">nrutman</a>, <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aboyko" class="user-hover" rel="aboyko">aboyko</a>, now that patch <a href="https://review.whamcloud.com/26088/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26088/</a> "<tt>mdc: allow setting readdir RPC size parameter</tt>" is landed, could someone at Cray run some testing to see what a better readdir RPC size is for the original workload? From <a href="#comment-58547" target="_blank" rel="noopener">comment-58547</a> it looks to be somewhere between 2 and 32 pages by default. We could start by reducing the default RPC size to something that gives good performance out of the box for most workloads.</p> <p>Separately, getting a patch to continue caching <tt>readdir()</tt> data on the client fd after lock cancellation (per <a href="#comment-58611" target="_blank" rel="noopener">comment-58611</a> and few replies) until <tt>close()</tt> or <tt>rewinddir()</tt> (I believe this ends up as <tt>seek(0)</tt> in the kernel) would allow even better client performance, since it would not need to re-fetch the directory for every deleted file.</p> <p>I will ask Eugene if he can take a look.</p> <p>See also <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a>, esp. patch <a href="https://review.whamcloud.com/#/c/7909/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/7909/</a></p> <p>Zam, this is the ticket discussed today that describes possible optimizations for single-threaded "<tt>rm -r</tt>" and other workloads that are doing mixed <tt>readdir() + unlink()</tt>.</p> Duplicate LU-5232 LU-4906 LU-4096 Related LU-10225 LU-1431 LU-5232 LU-15535 LU-1167 LU-9458 LU-10999 LU-11000 LU-17493 LU-8641 Development Rank 1|hzvqlz: Rank (Obsolete) 8195 Severity