https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6984 Lustre <p>In my testing of DNE2 I'm seeing problems when creating 1 million+ files per directory. Clearing out the debug logs I see the problem is only on the client side. When running a application I see:</p> <p>command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 100000 -i 5 -d /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test/shared_1000k_10<br/> Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test<br/> FS: 21.8 TiB Used FS: 0.2% Inodes: 58.7 Mi Used Inodes: 4.6%</p> <p>10 tasks, 1000000 files/directories<br/> aprun: Apid 3172: Caught signal Window changed, sending to application<br/> 08/03/2015 10:34:45: Process 0(nid00028): FAILED in create_remove_directory_tree, Unable to remove directory: No such file or directory<br/> Rank 0 <span class="error">&#91;Mon Aug 3 10:34:45 2015&#93;</span> <span class="error">&#91;c0-0c0s1n2&#93;</span> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0<br/> _pmiu_daemon(SIGCHLD): <span class="error">&#91;NID 00028&#93;</span> <span class="error">&#91;c0-0c0s1n2&#93;</span> <span class="error">&#91;Mon Aug 3 10:34:45 2015&#93;</span> PE RANK 0 exit signal Aborted<br/> aprun: Apid 3172: Caught signal Interrupt, sending to application<br/> _pmiu_daemon(SIGCHLD): <span class="error">&#91;NID 00012&#93;</span> <span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;Mon Aug 3 10:50:50 2015&#93;</span> PE RANK 7 exit signal Interrupt<br/> _pmiu_daemon(SIGCHLD): <span class="error">&#91;NID 00018&#93;</span> <span class="error">&#91;c0-0c0s6n2&#93;</span> <span class="error">&#91;Mon Aug 3 10:50:50 2015&#93;</span> PE RANK 9 exit signal Interrupt<br/> _pmiu_daemon(SIGCHLD): <span class="error">&#91;NID 00013&#93;</span> <span class="error">&#91;c0-0c0s6n1&#93;</span> <span class="error">&#91;Mon Aug 3 10:50:50 2015&#93;</span> PE RANK 8 exit signal Interrupt</p> <p>After the test failed any attempt to remove the files create by these test fail. When I attempt to remove the files I see the following errors in dmesg.</p> <p>LustreError: 5430:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) Skipped 7 previous similar messages<br/> LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2</p> pre-2.8 clients with DNE2 directories which contain 1 million or more files. LU-6984

Failure to delete over a million files in a DNE2 directory.

Bug Critical Resolved Fixed Di Wang James A Simmons dne2 Tue, 11 Aug 2015 14:40:54 +0000 Mon, 20 Feb 2017 07:41:57 +0000 Sat, 3 Oct 2015 04:21:28 +0000 Lustre 2.8.0 Lustre 2.8.0 0 11 <p>I attached to this ticket the client logs from when it failed to delete some of the files in the DNE2 directory. Let me know if the logs are good enough. Please be aware that the deletion of files sometimes works so it is some race condition. The logs cover both failed and successful attempts to delete files.</p> <p>James, how many shards in the striped directory?</p> <p>Each file is striped across 56 OSTs and the problem exist for any MDS striping greater than one.</p> <p>James: I did not see error cases in the debug log? Will you attach another log? Thanks</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@testnode lustre-release_new]# grep -r "ll_prep_inode" /work/LU-6381.log 00000080:00000001:4.0:1438629890.366795:0:12535:0:(llite_lib.c:2252:ll_prep_inode()) Process entered 00000080:00000001:4.0:1438629890.366895:0:12535:0:(llite_lib.c:2319:ll_prep_inode()) Process leaving via out (rc=0 : 0 : 0x0) 00000080:00000001:4.0:1438629890.369895:0:12535:0:(llite_lib.c:2252:ll_prep_inode()) Process entered 00000080:00000001:4.0:1438629890.369977:0:12535:0:(llite_lib.c:2319:ll_prep_inode()) Process leaving via out (rc=0 : 0 : 0x0) 00000080:00000001:4.0:1438629892.079356:0:12597:0:(llite_lib.c:2252:ll_prep_inode()) Process entered 00000080:00000001:4.0:1438629892.079459:0:12597:0:(llite_lib.c:2319:ll_prep_inode()) Process leaving via out (rc=0 : 0 : 0x0) 00000080:00000001:4.0:1438629892.082521:0:12597:0:(llite_lib.c:2252:ll_prep_inode()) Process entered 00000080:00000001:4.0:1438629892.082605:0:12597:0:(llite_lib.c:2319:ll_prep_inode()) Process leaving via out (rc=0 : 0 : 0x0) </pre> </div></div> <p>Somethings it doesn't print that message when it fails. Do you see anything useful in the log?</p> <p>There are only successful cases in the log, so not very helpful</p> <p>Could you please tell me how to reproduce the problem? </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>After the test failed any attempt to remove the files create by these test fail. When I attempt to remove the files I see the following errors in dmesg. </pre> </div></div> <p>James: Could you please collect the client debug log (-1) when you tries to remove these files after failure? Thanks. </p> <p>WangDi here is another client dump. When enabling full debug the file deletion race becomes less common. I do see this in the log:</p> <p>00000080:00002000:1.0:1439485513.362411:0:32074:0:(dcache.c:181:ll_ddelete()) keeping dentry mdtest_tree.0 (ffff8803ed204e0<br/> 0, parent ffff8803ecd3c200, inode ffff8803ea2d2a40) hashed,<br/> 00000080:00000001:1.0:1439485513.362413:0:32074:0:(dcache.c:203:ll_ddelete()) Process leaving (rc=0 : 0 : 0)</p> <p>As for reproducing this you just need one client node. On the client node create a directory with the following.</p> <p>lfs setdirstripe -c 4 -i 5 dne2_4_mds_md_test<br/> lfs setdirstripe -c 4 -i5 -D dne2_4_mds_md_test<br/> lfs setstripe -c 56 dne2_4_mds_md_test</p> <p>So you end up with a directory striped across 56 OSTs and 4 MDTs starting at MDT 5. This is with one MDT per MDS server. The 56 OSTs are split evenly across 4 OSS servers. In total I have 16 MDS servers each with one MDT.<br/> One of the example mdtest runs failing is:</p> <p>mpirun -n 1 mdtest -I 1000000 -i 5 -d /lustre/sultan/stf008/scratch/jsimmons/<br/> dne2_4_mds_md_test/shared_1000k_1</p> <p>In this case it is running just one thread on one node. I split the file count for -l by the number of total client threads so the total is always about 1 million files.</p> <p>Recent test with striping across 16 MDS servers shows this problem with mdtest in every case. So this might be the best configuration for you to reproduce this error. It fails with even 10k per directory.</p> <p>James: unfortunately, there are still not much useful information in the lctldump log. Hmm, I checked the code again to see what can cause</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 5430:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int ll_prep_inode(struct inode **inode, struct ptlrpc_request *req, struct super_block *sb, struct lookup_intent *it) { ............ *inode = ll_iget(sb, cl_fid_build_ino(&amp;md.body-&gt;mbo_fid1, sbi-&gt;ll_flags &amp; LL_SBI_32BIT_API), &amp;md); if (IS_ERR(*inode)) { #ifdef CONFIG_FS_POSIX_ACL if (md.posix_acl) { posix_acl_release(md.posix_acl); md.posix_acl = NULL; } #endif rc = IS_ERR(*inode) ? PTR_ERR(*inode) : -ENOMEM; *inode = NULL; CERROR("new_inode -fatal: rc %d\n", rc); GOTO(out, rc); } } </pre> </div></div> <p>Then in ll_iget(), there are only 3 places which can cause this ENOENT error.</p> <p>1. ll_iget() --<del>&gt;ll_read_inode2()</del>&gt;ll_update_inode()--<del>&gt;ll_update_lsm_md()</del>-<del>&gt;ll_init_lsm_md()</del>--&gt;ll_iget_anon_dir() , and you should see sth like "failed get simple inode ...." on client console. Did you see that on client console after failure happens? Hmm, this is caused by the failure in iget_locked(), seems unlikely. <br/> 2. ll_iget()----&gt;cl_file_inode_init(), and then you should see sth like "Failure to initialize cl object ...." on client console, did you see that?<br/> 3. ll_iget()---<del>&gt;ll_update_inode()</del>&gt;..... then same as 1.</p> <p>So could you please post the client console log (dmesg output) when these failure happens? Thanks.</p> <p>I am not sure I can find an env to do such test, but I can try. Thanks.</p> <p>I'm not seeing those error <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I tried this on a cluster with 4 MDSs x 4 MDTs (16 MDTs), 2 OSS x 4 OSTs (8 OSTs). </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 lustre]# lfs setdirstripe -c 4 -i 5 test1 [root@c01 lustre]# lfs setdirstripe -c4 -i 5 -D test1 [root@c01 lustre]# lfs setstripe -c 8 test1 [root@c01 lustre]# ls test1 [root@c01 lustre]# mpirun -n 1 mdtest -I 1000000 -i 5 -d /mnt/lustre/test1/shared_1000k -- started at 08/26/2015 14:56:55 -- mdtest-1.8.3 was launched with 1 total task(s) on 1 nodes Command line used: mdtest -I 1000000 -i 5 -d /mnt/lustre/test1/shared_1000k Path: /mnt/lustre/test1 FS: 3.9 TiB Used FS: 0.1% Inodes: 58.5 Mi Used Inodes: 0.0% 1 tasks, 1000000 files/directories </pre> </div></div> <p>It has been more than 3 hours, and still running. James: how long did it take in your run? Did I do sth wrong here?</p> <p>It really does take that long <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Just update, I tried this several times with build 34198, and I can not reproduce the problem. My cluster has 4 MDSs x 4 MDTs (16 MDTs), 2 OSS x 4 OSTs (8 OSTs). </p> <p>James: could you please retry this on current master if this failure is still there?</p> <p>With the latest master and a few patches this problem still exist. The "ll_prep_inode" errors are gone now. I tried to debug it with lctl debug but turning on debug hides the race condition. Do you know what areas I should add old fashion printks to see what is causing the failure?</p> <p>Hmm, It seems there might be two places in mdtest.c, which can trigger the problem.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#else if (rmdir(dir) == -1) { FAIL("Unable to remove directory"); } #endif </pre> </div></div> <p>and </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (rmdir(temp_path) == -1) { FAIL("Unable to remove directory"); } </pre> </div></div> <p>So could you please add some debug information here to print out the dir or temp_path when these rmdir fail happens.</p> <p>So if rmdir() fails, on the kernel side, that is probably caused by two reasons.</p> <p>1. path resolution failed for "dir" or "temp_path", I am not sure there are good way to printk the error message, without changing the kernel. But let's see what are these directories are when the failure happens.</p> <p>2. rmdir fails in ll_rmdir(), for this failure, you can simply add a patch like this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--- a/lustre/llite/namei.c +++ b/lustre/llite/namei.c @@ -1198,7 +1198,10 @@ static int ll_rmdir(struct inode *dir, struct dentry *dchild) if (rc == 0) { ll_update_times(request, dir); ll_stats_ops_tally(ll_i2sbi(dir), LPROC_LL_RMDIR, 1); - } + } else { + CERROR("unlink fails :name=%.*s, dir="DFID"(%p) rc: %d\n", + name-&gt;len, name-&gt;name, PFID(ll_inode2fid(dir)), dir, rc); + } ptlrpc_req_finished(request); RETURN(rc); </pre> </div></div> <p>TBH, I suspect the failure is caused by 1. Sigh, I can not reproduce it locally. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Good news I can duplicate this problem on a non-Cray node. I'm debugging it. Will let you know what I find.</p> <p>The ll_prep_inode errors came back so I back track to see what was causing the error. From my cheesy debugging I found this:</p> <p>[ 251.338486] Lustre: looks like a bad stripe<br/> [ 251.342779] Lustre: lmv_revalidate_slaves failed<br/> [ 251.347499] Lustre: md_merge_attr failed to validate the lsm<br/> [ 251.353263] Lustre: ll_update_lsm_md failed when called from ll_update_inode<br/> [ 251.360412] Lustre: ll_update_inode return -2<br/> [ 251.364877] Lustre: ll_read_inode2 failed<br/> [ 251.369061] Lustre: Skipped 1 previous similar message<br/> [ 251.374321] LustreError: 18546:0:(llite_lib.c:2305:ll_prep_inode()) new_inode -fatal: rc -2</p> <p>So the problem is from lmv_revalidate_slaves. It's reporting back a ENOENT due to body-&gt;mbo_nlink &lt; 2 condition. The comment points to a race between close(unlink) and getattr.</p> <p>James, thanks for updating, very useful information. Do you happen to know who calls ll_prep_inode() ? Thanks!</p> <p>Better yet I did a dump_stack at where the code fails. You have complete back traces. I attached the file to this ticket.</p> <p>Could you please tell me how to reproduce the problem? still use mdtest with single thread on 1 node? thanks.</p> <p>Doesn't matter how many client nodes. I use 400 below but use whatever you want. What matters the number of files per directory. Remember this is with remote_dir=-1 and remote_dir_gid=-1. Try using 8 MDS servers but any number greater than 1 will do:</p> <p>lfs setdirstripe -c 8 /lustre/whatever/jsimmons/dne2_8_mds_md_test<br/> lfs setdirstripe -c 8 -D /lustre/whatever/jsimmons/dne2_8_mds_md_test (to make all directories under it the same)<br/> mkdir /lustre/whatever/jsimmons/dne2_8_mds_md_test/shared_1000k_400<br/> mpi_run -n 400 mdtest -I 2500 -i 5 -d /lustre/whatever/jsimmons/dne2_8_mds_md_test/shared_1000k_400</p> <p>When mdtest goes to delete the files mdtest will fail. At least it does for me.</p> <p>Hmm, during slaves revalidation, it seems the striped directory has been locked with both LOOKUP and UPDATE locks. I do not understand why the master stripe nlink turns to 1 at that time. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>James: Could you please collect the debug log when the failure happens? (-1) would be best, but if there is race, just collect the default one please. Thanks!</p> <p>On the MDS or the client?</p> <p>Both would be best. If not, then only client would be ok. Thanks</p> <p>Ok, I tried to reproduce it on Opensfs cluster with 8 MDTs (4 MDS) and 4 OSTs(2 OSS). 9 clients. Just start the test, it has been an hour, still can not see this problem. I will check tomorrow morning to see how it goes?</p> <p>James: could you please tell me all of your patches(based on master)? Thanks.</p> <p>I did you one better. Grab my source rpm at <a href="http://www.infradead.org/~jsimmons/lustre-2.7.59-1_g703195a.src.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://www.infradead.org/~jsimmons/lustre-2.7.59-1_g703195a.src.rpm</a></p> <p>James: thanks. And usually how soon did you met the failure? after a few minutes? a few hours after starting the test?</p> <p>The first run of mdtest takes a while before failure. Once it fails you can duplicate the failure with rm -rf the left over files from mdtest.</p> <p>I attached the logs for my latest test from the client node and the all the MDS servers I have. </p> <p>Ah, It seems this check in lmv_revalidate_slaves is not correct</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (unlikely(body-&gt;mbo_nlink &lt; 2)) { /* If this is bad stripe, most likely due * to the race between close(unlink) and * getattr, let's return -EONENT, so llite * will revalidate the dentry see * ll_inode_revalidate_fini() */ CDEBUG(D_INODE, "%s: nlink %d &lt; 2 bad stripe %d" DFID ":" DFID"\n", obd-&gt;obd_name, body-&gt;mbo_nlink, i, PFID(&amp;lsm-&gt;lsm_md_oinfo[i].lmo_fid), PFID(&amp;lsm-&gt;lsm_md_oinfo[0].lmo_fid)); if (it.d.lustre.it_lock_mode &amp;&amp; lockh) { ldlm_lock_decref_and_cancel(lockh, it.d.lustre.it_lock_mode); it.d.lustre.it_lock_mode = 0; } GOTO(cleanup, rc = -ENOENT); } </pre> </div></div> <p>Because </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* * The DIR_NLINK feature allows directories to exceed LDISKFS_LINK_MAX * (65000) subdirectories by storing "1" in i_nlink if the link count * would otherwise overflow. Directory tranversal tools understand * that (st_nlink == 1) indicates that the filesystem dose not track * hard links count on the directory, and will not abort subdirectory * scanning early once (st_nlink - 2) subdirs have been found. * * This also has to properly handle the case of inodes with nlink == 0 * in case they are being linked into the PENDING directory */ </pre> </div></div> <p>I will remove this.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16490" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16490</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6984" title="Failure to delete over a million files in a DNE2 directory." class="issue-link" data-issue-key="LU-6984"><del>LU-6984</del></a> lmv: remove nlink check in lmv_revalidate_slaves<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d676ea2ab1f55dfa8e04ed5fa074444315808329</p> <p>So far your fix seems to have resolved this issue. Tomorrow I will run the mdtest to see if everything works.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16490/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16490/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6984" title="Failure to delete over a million files in a DNE2 directory." class="issue-link" data-issue-key="LU-6984"><del>LU-6984</del></a> lmv: remove nlink check in lmv_revalidate_slaves<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5725559786086f328f4c899936967eb6e5dce46e</p> <p>I have tested this patch extensively and it has resolved this issue. Now that it has landed we can close this ticket.</p> <p>Right you are James! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Related LU-6831 Development Epic/Theme dne Rank 1|hzxk7z: Rank (Obsolete) 9223372036854775807 Severity