https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4906 Lustre <p>On 2.5.57-79-ge7f99a5 I see that rm -rf on a directory with n regular files causes O&#40;n&#41; MDS_READPAGE requests to be sent. I ran the following on 2.4.3 and master:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>export MOUNT_2=y llmount.sh cd /mnt/lustre echo clear | tee /proc/fs/lustre/mdc/*/stats mkdir d touch d/{0..255} cat /proc/fs/lustre/mdc/*/stats cd /mnt/lustre2 echo clear | tee /proc/fs/lustre/mdc/*/stats rm -rf d cat /proc/fs/lustre/mdc/*/stats </pre> </div></div> <p>On 2.4.3:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>## mkdir and touch req_waittime 773 samples [usec] 65 55988 339933 3295555639 req_active 773 samples [reqs] 1 1 773 773 mds_close 256 samples [usec] 65 486 58958 16078566 mds_reint 257 samples [usec] 93 826 74764 29250026 ldlm_enqueue 259 samples [usec] 191 1423 150223 115570903 seq_query 1 samples [usec] 55988 55988 55988 3134656144 ## rm -rf snapshot_time 1397509895.872324 secs.usecs req_waittime 258 samples [usec] 50 1054 32830 6165600 req_active 258 samples [reqs] 1 1 258 258 ldlm_cancel 258 samples [usec] 50 1054 32830 6165600 snapshot_time 1397509895.872354 secs.usecs req_waittime 524 samples [usec] 45 9854 212336 225452482 req_active 524 samples [reqs] 1 4 876 1660 mds_close 1 samples [usec] 390 390 390 152100 mds_reint 257 samples [usec] 331 9854 151751 209416267 mds_readpage 3 samples [usec] 271 323 902 272634 ldlm_enqueue 261 samples [usec] 45 736 59120 15595504 ldlm_cancel 2 samples [usec] 64 109 173 15977 </pre> </div></div> <p>On master:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>## mkdir and touch: snapshot_time 1397507941.992796 secs.usecs snapshot_time 1397507941.992828 secs.usecs req_waittime 1282 samples [usec] 50 2674 364172 203043372 req_active 1282 samples [reqs] 1 1 1282 1282 mds_close 256 samples [usec] 61 640 50251 15984775 mds_reint 257 samples [usec] 89 1045 60223 21230981 mds_getxattr 256 samples [usec] 50 658 38276 9345158 ldlm_enqueue 513 samples [usec] 82 2674 215422 156482458 ## rm -rf snapshot_time 1397507954.948995 secs.usecs req_waittime 991 samples [usec] 31 5949 371017 322109413 req_active 991 samples [reqs] 1 9 2132 6404 mds_close 1 samples [usec] 126 126 126 15876 mds_reint 257 samples [usec] 168 5949 173654 221727790 mds_readpage 132 samples [usec] 158 2173 44232 21316906 mds_getxattr 60 samples [usec] 31 345 5769 828911 ldlm_enqueue 423 samples [usec] 44 2496 123479 70146809 ldlm_cancel 118 samples [usec] 65 891 23757 8073121 snapshot_time 1397507954.949096 secs.usecs req_waittime 1 samples [usec] 108 108 108 11664 req_active 1 samples [reqs] 1 1 1 1 ldlm_cancel 1 samples [usec] 108 108 108 11664 </pre> </div></div> <p>(If you noticed that ldlm_enqueue and mds_reint are present, it's because I used <a href="http://review.whamcloud.com/#/c/6223/1" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6223/1</a> which is awesome and has been landable for nearly one year (still on patch set 1), but nobody ever reviews it.)</p> LU-4906

rm -rf triggers too much MDS_READPAGE

Bug Critical Resolved Fixed Di Wang John Hammond llite mdc readdir Mon, 14 Apr 2014 21:24:23 +0000 Thu, 19 Jun 2014 19:44:41 +0000 Thu, 19 Jun 2014 17:33:21 +0000 Lustre 2.6.0 Lustre 2.6.0 0 8 <p>This is partly related to <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a>, which is causing too-large readdir() RPCs to be issued and slow down the rm process. It would be allowed for the readdir() of the rm process to cache the entries and not issue more RPCs, either in userspace or in the kernel.</p> <p>I've never looked at the rm implementation, but I imagine it is complex by its very presence in GNU fileutils. </p> <p><a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a> also describes the mechanism by which the kernel could cache the directory entries or pages on the file descriptor in the kernel even after the MDS revoked the DLM lock. </p> <p>Potentially this is also related to <a href="https://jira.whamcloud.com/browse/LU-4367" title="unlink performance regression on lustre-2.5.52 client" class="issue-link" data-issue-key="LU-4367"><del>LU-4367</del></a></p> <p>I think rm was always redoing readdir after every unlink, we had a ticket about this in bugzilla even that I can find if pressed.<br/> This is all thanks to us not implementing readdir caching on per fd basis, so every time you rm something, the readdir cache is invalidated and next readdir leads to lock refetch.<br/> There's some glibc caching (that does getdirents64), but apparently it's no longer works or something, no idea why ad it's not affected by lustre at all.</p> <p>On 2.4.2 (FC12 userspace) I see strace only does a single readdir:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>strace rm -r /myth/tmp/ttt : : getdents(3, /* 10 entries */, 32768) = 240 newfstatat(3, "2", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 stat("/myth/tmp/ttt/2", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 access("/myth/tmp/ttt/2", W_OK) = 0 unlinkat(3, "2", 0) = 0 newfstatat(3, "7", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 stat("/myth/tmp/ttt/7", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 access("/myth/tmp/ttt/7", W_OK) = 0 unlinkat(3, "7", 0) = 0 newfstatat(3, "4", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 stat("/myth/tmp/ttt/4", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 access("/myth/tmp/ttt/4", W_OK) = 0 unlinkat(3, "4", 0) = 0 newfstatat(3, "3", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 stat("/myth/tmp/ttt/3", {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 access("/myth/tmp/ttt/3", W_OK) = 0 </pre> </div></div> <p>What is the userspace being tested here, and is it the same for both tests? What does the strace show for getdents() calls?</p> <p>so I just run strace on rm on lustre (current master) and I see getdents64 caching on the order of 1000 entries per call, btw.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>getdents(3, /* 1001 entries */, 32768) = 32024 brk(0x15cf000) = 0x15cf000 brk(0x15f0000) = 0x15f0000 getdents(3, /* 0 entries */, 32768) = 0 close(3) = 0 unlinkat(4, "file221", 0) = 0 ... </pre> </div></div> <p>Oleg, I should have put more detail here. I was running strace alongside and rm -rf calls getdents twice, unlink 256 times, and rmdir once, it that order. It's the statahead thread that generates all but 2-3 of the readpage requests.</p> <p>ok, so it's statahead's fault, I see.</p> <p>Fan Yong, it seems that statahead is interacting badly with "rm -r". Is there some way to stop it from trying to prefetch the directory entries if the directory lock is being cancelled too often?</p> <p>The statahead behaviour has been changed after moving the directory page cache from LLITE to MDC for striped directory.</p> <p>Originally, the directory page cache is in LLITE, and the ldlm lock is only held when fetch the directory page from MDS. After that the directory ldlm lock will be released. And then the statahead thread will traversal the directory page without the directory ldlm lock, even if someone is performing "rm -rf" and cancel (or ELC) the directory ldlm lock from the client, the statahead still holds the page reference. It is not a serious issue for statahead to get "-ENOENT" because of some name entry has been removed, because it is statahead internal failure and invisible to the applications. The statahead thread still can go ahead to fee-fetch as much as possible. So there is no much directory ldlm lock ping-pong.</p> <p>But the situation has changed after moving the directory page to MDC. In current implementation, when the statahead traversal the directory, it does not hold the page reference, it needs to verify the directory ldlm lock for every name entry. That means there will be the directory ldlm lock ping-pong between the statahead thread and the "rm -rf" thread for every name entry.</p> <p>Summary, the original directory ldlm lock ping-pong is per page, the current directory ldlm lock ping-pong is per name entry. That is why the bad performance.</p> <p>Currently, the LLITE cannot be aware of the directory ldlm lock ping-pong which happened in MDC, so the statahead cannot know it should stop itself because of the bad performance unless someone indicates that.</p> <p>Di, it seems this is caused by the DNE2 readdir changes. Could you please look at this c</p> <p>I think the easiest and best solution here is to have llite cache the readdir directory pages on the file handle if the lock is cancelled, either until the file handle is closed or until it does rewinddir(). That avoids extra READPAGE RPCs for many reasons.</p> <p>From the readdir() and rewinddir() specifications in SUSv2: <a href="http://pubs.opengroup.org/onlinepubs/007908799/xsh/rewinddir.html" class="external-link" target="_blank" rel="nofollow noopener">http://pubs.opengroup.org/onlinepubs/007908799/xsh/rewinddir.html</a></p> <blockquote> <p>If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir_r() returns an entry for that file is unspecified.</p> <p>The readdir_r() function may buffer several directory entries per actual read operation; the readdir_r() function marks for update the st_atime field of the directory each time the directory is actually read.</p> <p>The rewinddir() function resets the position of the directory stream to which dirp refers to the beginning of the directory. It also causes the directory stream to refer to the current state of the corresponding directory, as a call to opendir() would have done.</p></blockquote> <p>Hmm, for non-striped directory, it does not matter cached it on llite or mdc, IMHO. But for striped directory, caching the dir-entry page in MDC layer would help LMV to build the dir-entry page by hash-order from all of stripes.</p> <p>In current readdir implementation, it iterate the dir_entry one by one(i.e. go through the stack for each entry), which is the reason causes this performance issue. I am working on a patch to build the dir-entry page by page in LMV layer, which should resolve the issue. There are still some issue with the patch, I will post the patch soon.</p> <p>With this patch <a href="http://review.whamcloud.com/#/c/10622/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10622/</a><br/> rm -rf dir with 10k files goes from 64 seconds to 8 seconds in my local test,<br/> with the patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@mds tests]# ./createmany -o /mnt/lustre/test1/f- 10000 total: 10000 creates in 11.72 seconds: 853.50 creates/second [root@mds tests]# time rm -rf /mnt/lustre/test1 real 0m8.151s user 0m0.012s sys 0m2.242s </pre> </div></div> <p>without patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@mds tests]# ./createmany -o /mnt/lustre/test1/f- 10000 total: 10000 creates in 11.98 seconds: 835.02 creates/second [root@mds tests]# ls /mnt/lustre/test1/ | wc 10000 10000 68890 [root@mds tests]# time rm -rf /mnt/lustre/test1 real 1m4.496s user 0m0.014s sys 0m3.832s </pre> </div></div> <p>This kind of performance test results should be great to include in the patch commit message.</p> <p>I added another patch <a href="http://review.whamcloud.com/10695" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10695</a> based on 10622, to add striped entry page cache to master inode, please check.</p> <p>Patch landed to Master. Remaining patch will be tracked under a new ticket.</p> <p>I very strongly suspect the landing of this fix has broken lustre builds on 3.12 kernels. See comment in <a href="https://jira.whamcloud.com/browse/LU-4416" title="support for 3.12 linux kernel" class="issue-link" data-issue-key="LU-4416"><del>LU-4416</del></a>.</p> Duplicate LU-3308 Related LU-4367 LU-4902 LU-5232 Development Rank 1|hzwk33: Rank (Obsolete) 13548 Severity