https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-762 Lustre <p>mdtest in shared directory fails with 50 clients and MDTEST_NFILES=1024, also fails w/100 clients and MDTEST_NFILES=256. Failure is silent, no Lustre Errors or any servers messages. Failure is consistent and repeatable. Failure only occurs when run in shared directory mode. <br/> Failure example:</p> <p>000: Command line used: /opt/mdtest-1.8.3/bin/mdtest -d /p/l_wham/white215/hyperion.3867/mdtest -i3 -n1024<br/> 000: Path: /p/l_wham/white215/hyperion.3867<br/> 000: FS: 87.0 TiB Used FS: 0.0% Inodes: 546.9 Mi Used Inodes: 2.1%<br/> 000:<br/> 000: 400 tasks, 409600 files/directories<br/> 000: 10/14/2011 01:17:15: Process 0(hyperion319): FAILED in create_remove_items_helper, unable to remove directory: No such file or directory<br/> srun: mvapich: 2011-10-14T01:17:15: ABORT from MPI rank 0 <span class="error">&#91;on hyperion319&#93;</span><br/> 000: <span class="error">&#91;0&#93;</span> <span class="error">&#91;MPI Abort by user&#93;</span> Aborting Program!<br/> 000: <span class="error">&#91;0:hyperion319&#93;</span> Abort: MPI_Abort() code: 1, rank 0, MPI Abort by user Aborting program ! at line 99 in file mpid_init.c<br/> 000: slurmd<span class="error">&#91;hyperion319&#93;</span>: *** STEP 1219936.0 KILLED AT 2011-10-14T01:17:15 WITH SIGNAL 9 ***</p> RHEL5/x86_64 LU-762

Hyperion - mdtest failure

Bug Major Resolved Fixed Yang Sheng Cliff White Fri, 14 Oct 2011 08:59:54 +0000 Tue, 10 Jan 2012 17:46:41 +0000 Fri, 16 Dec 2011 14:11:24 +0000 Lustre 1.8.7 Lustre 2.2.0 Lustre 2.1.1 0 6 <p>I tried to reproduce this bug on Toro, it can pass with 3 clients and MDTEST_NFILES=1024, but failed with 3 clients and MDTEST_NFILES=10240:</p> <p>MDTEST_NFILES=1024:<br/> ----------------------<br/> mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 12288 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 6095.885 4288.836 5074.051 756.460<br/> Directory stat : 28560.081 26921.815 27471.125 770.018<br/> Directory removal : 4370.147 2651.794 3363.277 731.949<br/> File creation : 7989.778 7864.541 7912.732 55.049<br/> File stat : 23978.178 22347.507 23185.891 666.516<br/> File removal : 8990.494 8197.678 8721.670 370.560<br/> Tree creation : 2770.346 20.672 1548.695 1143.263<br/> Tree removal : 1261.066 16.791 628.840 508.173</p> <p>&#8211; finished at 10/14/2011 16:45:06 &#8211;<br/> <span class="error">&#91;root@client-15 ~&#93;</span># </p> <p>MDTEST_NFILES=10240:<br/> ---------------------------<br/> mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 122880 files/directories<br/> 10/14/2011 16:49:14: Process 2(client-2.lab.whamcloud.com): FAILED in create_remove_items_helper, unable to remove directory: No such file or directory<br/> --------------------------------------------------------------------------<br/> MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD <br/> with errorcode 1.</p> <p>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br/> You may or may not see output from other processes, depending on<br/> exactly when Open MPI kills them.<br/> --------------------------------------------------------------------------<br/> --------------------------------------------------------------------------<br/> mpirun has exited due to process rank 2 with PID 3690 on<br/> node client-2 exiting without calling "finalize". This may<br/> have caused other processes in the application to be<br/> terminated by signals sent by mpirun (as reported here).<br/> --------------------------------------------------------------------------<br/> <span class="error">&#91;root@client-15 ~&#93;</span># </p> <p>Oleg will look into this one</p> <p>Ok, so I was diving into it and it seems to be related to some ldiskfs problems.</p> <p>Here is a relevant mds log part:<br/> 00000100:00000001:23:1319040202.309449:0:6653:0:(events.c:322:request_in_callback()) Process leaving<br/> 00000004:00000001:10:1319040202.309449:0:7050:0:(mds_reint.c:1607:mds_get_parent_child_locked()) Process entered<br/> 00000004:00000001:10:1319040202.309450:0:7050:0:(mds_reint.c:92:mds_lookup()) Process entered00000004:00000001:10:1319040202.309454:0:7050:0:(mds_reint.c:115:mds_lookup()) Process leaving (rc=18446604446544310608 : -139627165241008 : ffff8102846a3150)<br/> 00000004:00000001:10:1319040202.309457:0:7050:0:(handler.c:256:mds_fid2dentry()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe)00000004:00000001:10:1319040202.309459:0:7050:0:(mds_reint.c:1614:mds_get_parent_child_locked()) Process leaving (rc=18446744073709551614 : -2 : fffffffffffffffe)</p> <p>This tells us that ldiskfs somehow returned an inode from lookup that had zero nlink which is not right at all, as nobody could delete it yet.</p> <p>So after the job dies I located this dir (from client logs) and did ls on it - it was still there, and then removed it - and it removed with no fuss.</p> <p>Another intereting point, in my testing I am only able to reproduce the problem if I run the two mdtests sequentially like specified here - one with 1024 files followed by 10240 files in the same dir.</p> <p>If I run just the 10240 files run in another dir or same dir after it was deleted, I cannot seem to be able to reproduce it, so I wonder if there is some leftover state in the large dir left that runins the day later on?</p> <p>Has anyone tried running this same test with 1.8.6 on the MDS to see if it passes? It would also be useful to either compile the 1.8.6 Lustre against the 2.6.18-274.3.1.el5 (RHEL 5) kernel from 1.8.7, or 1.8.7 against the 2.6.18-238.12.1.el5 (RHEL 5) from 1.8.6.</p> <p>It is possible that this is either an upstream ldiskfs bug, or possibly an existing bug that we didn't notice before due to a different test environment.</p> <p>I asked Sarah to try on 1.8.5 an 1.8.6 to see if it happens too.</p> <p>I finished the same tests on 1.8.6, run 4 times, they all pass:</p> <p><span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> connect to address 10.10.4.5 port 544: connect to address 10.10.4.2 port 544: Connection refused<br/> Connection refused<br/> Trying krb4 rsh...<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 13:01:08 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.3% Inodes: 2.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 12288 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 6351.096 3853.853 4965.091 1037.843<br/> Directory stat : 24382.547 14461.591 19932.550 4114.050<br/> Directory removal : 3375.762 2219.832 2795.249 471.920<br/> File creation : 7368.815 6689.622 7084.639 288.143<br/> File stat : 23931.050 23541.947 23698.468 167.708<br/> File removal : 8466.017 7895.573 8205.915 235.577<br/> Tree creation : 2369.663 213.028 1297.122 880.481<br/> Tree removal : 1602.715 80.465 631.532 688.807</p> <p>&#8211; finished at 10/19/2011 13:01:42 &#8211;<br/> <span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 13:02:24 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.3% Inodes: 2.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 122880 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 3407.669 2985.174 3263.238 196.670<br/> Directory stat : 19586.197 16856.908 18567.720 1217.032<br/> Directory removal : 2116.226 2032.805 2070.180 34.604<br/> File creation : 6603.196 6245.869 6459.676 154.112<br/> File stat : 22034.710 21846.257 21921.526 81.473<br/> File removal : 7017.384 6581.323 6831.171 183.620<br/> Tree creation : 1501.720 17.186 972.647 676.912<br/> Tree removal : 265.883 14.524 109.538 111.405</p> <p>&#8211; finished at 10/19/2011 13:09:44 &#8211;<br/> INIT: version 2.86 reloading<br/> <span class="error">&#91;root@client-15 ~&#93;</span># rpm -qa|grep lustre<br/> lustre-client-tests-1.8.6-wc1_2.6.18_238.12.1.el5<br/> lustre-client-1.8.6-wc1_2.6.18_238.12.1.el5<br/> lustre-client-modules-1.8.6-wc1_2.6.18_238.12.1.el5</p> <p>As Peter and Andreas suggested, besides 1.8.6, I tried two builds between 1.8.6 and 1.8.7-rc1--build #130 and #131. For #131, kernel is updated from 2.6.18-238.12.1.el5 to 2.6.18-274.3.1.el5. The result is #130 can pass both mdtest(1024 and 10240) while #131 hit this problem and after this failure, test with MDTEST_NFILES=1024 cannot pass either.</p> <p>#130<br/> ---------------------------------------------------------------------------<br/> <span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 17:19:32 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 12288 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 4853.888 3735.464 4346.004 462.328<br/> Directory stat : 16579.909 13008.459 15348.596 1655.482<br/> Directory removal : 2295.923 2191.987 2229.765 46.938<br/> File creation : 6724.576 6253.199 6528.509 200.431<br/> File stat : 23266.734 22682.011 23025.525 249.433<br/> File removal : 8424.377 7434.221 7924.369 404.290<br/> Tree creation : 2605.158 1328.152 1771.459 589.910<br/> Tree removal : 634.923 46.341 366.383 243.032</p> <p>&#8211; finished at 10/19/2011 17:20:12 &#8211;<br/> <span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 17:29:59 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 122880 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 3336.135 2802.707 3099.313 221.836<br/> Directory stat : 19458.564 16126.399 17976.454 1385.008<br/> Directory removal : 2532.999 2463.258 2490.151 30.625<br/> File creation : 6323.514 6055.286 6199.546 110.440<br/> File stat : 21670.612 21482.912 21568.812 77.449<br/> File removal : 7048.993 6775.570 6876.685 122.451<br/> Tree creation : 1633.932 31.719 1051.878 723.751<br/> Tree removal : 33.018 10.923 19.498 9.675</p> <p>&#8211; finished at 10/19/2011 17:36:58 &#8211;<br/> INIT: version 2.86 reloading</p> <p><span class="error">&#91;root@client-15 ~&#93;</span># rpm -qa|grep lustre<br/> lustre-client-tests-1.8.6.80-wc1_2.6.18_238.12.1.el5_geb7f696<br/> lustre-client-1.8.6.80-wc1_2.6.18_238.12.1.el5_geb7f696<br/> lustre-client-modules-1.8.6.80-wc1_2.6.18_238.12.1.el5_geb7f696<br/> <span class="error">&#91;root@client-15 ~&#93;</span># </p> <p>#131<br/> -----------------------------------------------------------------------<br/> <span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 21:52:55 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n1024<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 12288 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 5487.495 3353.100 4100.389 981.813<br/> Directory stat : 27125.824 25855.645 26577.083 532.733<br/> Directory removal : 7818.470 6933.606 7367.622 361.440<br/> File creation : 7796.005 7057.546 7418.017 301.729<br/> File stat : 24489.842 22745.956 23741.504 733.084<br/> File removal : 8910.799 8573.264 8721.124 140.935<br/> Tree creation : 3460.647 77.585 1758.095 1381.217<br/> Tree removal : 1204.567 750.726 1011.358 191.316</p> <p>&#8211; finished at 10/19/2011 21:53:21 &#8211;<br/> <span class="error">&#91;root@client-15 ~&#93;</span># mpirun -np 12 -H client-15,client-5,client-2 -mca plm_rsh_agent rsh /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> Trying krb4 rsh...<br/> connect to address 10.10.4.5 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> connect to address 10.10.4.2 port 544: Connection refused<br/> trying normal rsh (/usr/bin/rsh)<br/> &#8211; started at 10/19/2011 21:53:31 &#8211;</p> <p>mdtest-1.8.3 was launched with 12 total task(s) on 3 nodes<br/> Command line used: /usr/bin/mdtest -d /mnt/lustre/mdtest -i3 -n10240<br/> Path: /mnt/lustre<br/> FS: 46.9 GiB Used FS: 4.4% Inodes: 3.0 Mi Used Inodes: 0.0%</p> <p>12 tasks, 122880 files/directories<br/> 10/19/2011 21:54:18: Process 4(client-5.lab.whamcloud.com): FAILED in create_remove_items_helper, unable to remove directory: No such file or directory<br/> --------------------------------------------------------------------------<br/> MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD <br/> with errorcode 1.</p> <p>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br/> You may or may not see output from other processes, depending on<br/> exactly when Open MPI kills them.<br/> --------------------------------------------------------------------------<br/> --------------------------------------------------------------------------<br/> mpirun has exited due to process rank 4 with PID 3683 on<br/> node client-5 exiting without calling "finalize". This may<br/> have caused other processes in the application to be<br/> terminated by signals sent by mpirun (as reported here).<br/> --------------------------------------------------------------------------<br/> <span class="error">&#91;root@client-15 ~&#93;</span># rpm -qa|grep lustre<br/> lustre-client-tests-1.8.6.80-wc1_2.6.18_274.3.1.el5_gbec8184<br/> lustre-client-1.8.6.80-wc1_2.6.18_274.3.1.el5_gbec8184<br/> lustre-client-modules-1.8.6.80-wc1_2.6.18_274.3.1.el5_gbec8184<br/> <span class="error">&#91;root@client-15 ~&#93;</span># </p> <p>YangSheng</p> <p>Could you please look into what about the checkin identified as the culprit has caused this problem?</p> <p>thanks</p> <p>Peter</p> <p>Ok. Taken.</p> <p>Yang Sheng, can you please attach a diff between those two kernels fs/<b>.<span class="error">&#91;ch&#93;</span> fs/ext4/</b>.<span class="error">&#91;ch&#93;</span>. It may be that this is a change in the ext4 code and/or VFS that is causing the problem.</p> <p>fs diff</p> <p>ext4 diff</p> <p>I don't see any obvious changes in those two patches that might be causing this problem.</p> <p>In the ext4 code there is a patch in ext4_free_inode() that is duplicating our own ext4-claim_inode-free_inode-race.patch, which has been removed from the ldiskfs-2.6-rhel5-ext4.series. The rest of the patches are for AIO with O_DIRECT files, which is completely irrelevant for the MDS, and a small fix to the quota code.</p> <p>There is one change in the VFS code in d_obtain_alias(), but I don't see that we use this on the MDS. Possibly the problem relates to our iopen patch? I can't see anything obvious.</p> <p>Hmm, I thought the two builds being tested were only before/after the kernel update. I see that there is another patch (bec818434c27bb390b4c8866e73d1afb0dd9e884, <a href="http://review.whamcloud.com/1259" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1259</a>) that was also included in that patch, and it seems very relevant, and a good candidate for the cause of this bug.</p> <p>That patch was from <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a>.</p> <p>Actually I kind of don't think the <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> patch is relevant.</p> <p>We only see the problem in mkdir/rmdir case.<br/> The mkdir path is completely untouched by the patch.<br/> The rmdir path is touched a little bit - the commong get_parent_child_locked function is changed, but htere is "op" argument added and all new functionality is added after check for open as I see it.</p> <p>Besides, the erroneous lookup (rather, invalid nlink value) clearly comes from ldiskfs, and ldiskfs was not touched by <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a>.</p> <p>Can you please elaborate?</p> <p>btw I would agree that running a test with just kernel different and perhaps just lu-146 applied would be a good idea to clear who's at fault.</p> <p>btwm another note.<br/> See that hte failure was in trying to lookup paremt, tihs part of the mds_get_parent_locked() - at the very beginning that was not touched:</p> <p> /* Step 1: Lookup parent */<br/> *dparentp = mds_fid2dentry(mds, fid, NULL);<br/> if (IS_ERR(*dparentp)) </p> { rc = PTR_ERR(*dparentp); *dparentp = NULL; RETURN(rc); } <p>I will test the review build which contain one patch at a time. Will keep you updated.</p> <p>I tested review build #2398 which included patch for <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> and #2437 for <a href="https://jira.whamcloud.com/browse/LU-674" title="Update RHEL5 kernel to 2.6.18-274.3.1.el5 security update" class="issue-link" data-issue-key="LU-674"><del>LU-674</del></a>, got the same issue on both builds. I will rerun the tests on b1_8 build #130(doesn't contain these 2 patches)again to confirm that it can pass.</p> <p>From ext4_iget() (called from iopen):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (inode-&gt;i_nlink == 0) { if (inode-&gt;i_mode == 0 || !(EXT4_SB(inode-&gt;i_sb)-&gt;s_mount_state &amp; EXT4_ORPHAN_FS)) { /* this inode is deleted */ ret = -ESTALE; goto bad_inode; } /* The only unlinked inodes we let through here have * valid i_mode and are being read by the orphan * recovery code: that's fine, we're about to complete * the process of deleting those. */ } </pre> </div></div> <p>So the fs has EXT4_ORPHAN_FS set? How is this possible?<br/> The only option is that someone else dropped nlink after ext4_iget() successfully completed. This would thus be a race, so i would really not disregard (as discussed when this bug was initially reported) <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> which touches the locking code, although i have no idea how it could trigger this.</p> <p>Note that nlink=0 is reported on parent dir.<br/> The parent dir is never deleted, but it's nlink count is 1 (because we went past 32k dirs in it), so if some code does nlink--, we'll have 0 at least briefly.<br/> Also note that the place where we do lookup is not locked by ldlm lock yet, so if there is some possibility for parallel rmdir.</p> <p>Regarding the code you quote, I believe it's jsut that i_mode is NOT 0 (of course) and the EXT4_ORPHAN_FS is NOT set as well.</p> <p>So time to see what might be doing nlink-- on rmdir path even if briefly?</p> <p>Yes, I see this now - the directory link count is temporarily set to 0 if a subdirectory is deleted:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int ext4_rmdir(struct inode *dir, struct dentry *dentry) { : : ext4_dec_count(handle, dir); : : } static void ext4_dec_count(handle_t *handle, struct inode *inode) { drop_nlink(inode); if (S_ISDIR(inode-&gt;i_mode) &amp;&amp; inode-&gt;i_nlink == 0) inc_nlink(inode); } </pre> </div></div> <p>This is called via ll_vfs_rmdir(). In mds_reint_create() and mds_mfd_close it calls ll_vfs_rmdir(dir) under LOCK_INODE_MUTEX(dir). Perhaps we are not doing the lookup under i_mutex, or not refreshing the attributes under i_mutex?</p> <p>It looks like the patch landed to ext4 is not the same as what was in our original patch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static inline void ext3_dec_count(handle_t *handle, struct inode *inode) { if (!S_ISDIR(inode-&gt;i_mode) || inode-&gt;i_nlink &gt; 2) drop_nlink(inode); } </pre> </div></div> <p>I don't know if the locking changes in <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> changed this in some way.</p> <p><a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> did not change any locking outside of the open path. Also it had no ldiskfs changes too.</p> <p>What I meant was whether the changes in the <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a> locking made it racy w.r.t. getting the i_nlink count for the parent directory without doing LOCK_INODE_MUTEX(), so that if another process is doing ext4_dec_count() on the parent at that same time it can briefly see the parent directory with i_nlink == 0.</p> <p>It is possible to add another ext4 patch to change ext4_dec_count() back to how it was in our patches, which is probably the most expedient thing to do for 1.8.7, but it would be better to avoid the race in Lustre instead of adding more patches to ldiskfs. That said, I don't think this is needed for 2.x, so maybe that is OK until 1.8.x is gone.</p> <p>I don't think lu-146 touches anything related to attrs readign and inude mutex around it, its likely a problem we had from the very beginning.</p> <p>BTW having that said, why would it be fine in the kernel to have nlink 0 even temporarily? It's error prone and all, why not have it correct at all times instead?<br/> This is something I think should also be addressed upstream.</p> <p>It is OK for the kernel because this is always accessed under i_mutex in cases that matter? I haven't checked yet, but my guess is that mds_get_parent_child_locked() (or somewhere else) is accessing i_nlink without a lock?</p> <p>That said, now I'm a bit confused because the code in question (ext4_iget()) is only hitting this code path when reading the inode from disk. That means either we are getting inode aliases in memory (seems unlikely, since it would be getting a valid i_nlink from disk, since ext4_dec_count() is only working on the in-memory inode) or we are accessing the parent inode before it is fully set up. It may possibly relate to strangeness in the iopen code, I haven't looked at that in a long time.</p> <p>It may be that my source of the i_nlink=0 condition is misdiagnosed.</p> <p>Your source of the problem seems to be very plausible.</p> <p>I am not sure what iget has to do with this, as we are getting this inode via normal lookup, we don't get any sort of an error from ldiskfs itself, just our own checking of nlink is finding that it's suddenly 0.</p> <p>And yes, we do check it without any locking:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> result = mds_lookup(obd, fid_name, mds-&gt;mds_fid_de, strlen(fid_name)); if (IS_ERR(result)) RETURN(result); inode = result-&gt;d_inode; if (!inode) RETURN(ERR_PTR(-ENOENT)); if (inode-&gt;i_nlink == 0) { if (inode-&gt;i_mode == 0 &amp;&amp; LTIME_S(inode-&gt;i_ctime) == 0 ) { LCONSOLE_WARN("Found inode with zero nlink, mode and " "ctime -- this may indicate disk" "corruption (device %s, inode %lu, link:" " %lu, count: %d)\n", obd-&gt;obd_name, inode -&gt;i_ino, (unsigned long)inode-&gt;i_nlink, atomic_read(&amp;inode-&gt;i_count)); } dput(result); RETURN(ERR_PTR(-ENOENT)); } </pre> </div></div> <p>I've just created <a href="http://review.whamcloud.com/1579" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1579</a> that has a patch which may fix this problem. It hasn't been tested at all, but hopefully it is right.</p> <p>I will test it when Toro is accessible.</p> <p>I've tested patch <a href="http://review.whamcloud.com/#change,1579" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1579</a> for 3 times, both 1024 and 10240 pass.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,ofa #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el6,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,ofa #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el6,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,ofa #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,ofa #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/148/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,inkernel #148</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=9500ebf6146dcd50dab094b502d4b50d0123296a" class="external-link" target="_blank" rel="nofollow noopener">9500ebf6146dcd50dab094b502d4b50d0123296a</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>The test fails on RC2 also:</p> <p>0000: mdtest-1.7.3 was launched with 1032 total task(s) on 129 nodes<br/> 0000: Command line used: /opt/mdtest-1.7.4/bin/mdtest -d /p/l_wham/d0.mdtest -i 3 -n 242 -u<br/> 0000: Path: /p/l_wham<br/> 0000: FS: 87.0 TiB Used FS: 0.0% 0.0%Inodes: 546.9 Mi Used Inodes: 0.0%<br/> 0000:<br/> 0000: 1032 tasks, 249744 files/directories<br/> 0746: 10/28/2011 00:24:25: Process 746(hyperion412): FAILED in unique_dir_access, Unable to create unique directory: No such file or directory<br/> 0831: 10/28/2011 00:24:25: Process 831(hyperion422): FAILED in unique_dir_access, Unable to create unique directory: No such file or directory</p> <p>This seems to be somewhat different problem now, fails in create because file does not exist? Need some logs I guess, probably from mds. e.g. +vfstrace +dlmtrace +...trace (this last one is the one that traces execution, though I forgot what the name is).</p> <p>I re-ran the test with the requested debug options, it failed. Am uploading to maloo now.</p> <p>Malloo results <a href="https://maloo.whamcloud.com/test_sessions/02a44c1a-01a0-11e1-bb4f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/02a44c1a-01a0-11e1-bb4f-52540025f9af</a></p> <p>MDS debug log</p> <p>Ok, this log is way too small, it only covers 9 directory creates out of 1038.</p> <p>I guess to get more consistent results, can you please increase debug_mb to a gig on clients annd to 2G on the MDS, in addition please use debug_daemon (with 10240 M argument to it) on the MDS.</p> <p>Once failed, capture the MDS log from the debug daemon, and also capture a couple of lustre debug logs from clinets that reported uniq create failures.</p> <p>I also checked mdtest 1.7.3 source and it appears the failure to create is the failure to create uniq dirs themselves, not during in the middle of the test surprisingly.</p> <p>Ran second test for Oleg Results at: <a href="https://maloo.whamcloud.com/test_sessions/c4849490-04af-11e1-aa99-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/c4849490-04af-11e1-aa99-52540025f9af</a><br/> Debug logs set to 2gb, which was too big for maloo, logs are on brent/eric ~cliffw/lu762<br/> Need instructions for next steps/test runs</p> <p>So, my understanding is from talking to Oleg and Cliff that the reported "reoccurence" has been linked to using an old and unreliable version of mdtest so this issue is actually fixed in 1.8.7 RC2.</p> <p>So, can this ticket be marked as resolved or does an equivalent fix need to land to master?</p> <p>I suspect we need this on master for the case of running on an upgraded fs that was originally started as 1.8.x formatted, so no oi is used.</p> <p>somebody need to confirm my understanding of this fact.</p> <p>Talking to Andreas it seems safest to port this change to master. YangSheng can you please take care of that? </p> <p>I'm not sure what you are asking about related to the OI. When a 1.8.x formatted filesystem is upgraded to 2.x it will get an OI to hold new FIDs, but old IGIF FIDs are not added to the OI (at least currently though it may make life easier in some cases, but that is a separate discussion).</p> <p>I think it is reasonable to land the ldiskfs patch to master as well, just to keep the ldiskfs patches in sync. The patch was already accepted upstream for ext4, so eventually the patch can go away.</p> <p>Yang Sheng, can you please add the ext4-nlink-2.6-rhel5.patch to the master ldiskfs series as well.</p> <p>Hi, Peter, Adilger, patch upload to:<a href="http://review.whamcloud.com/#change,1644" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1644</a></p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/350/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #350</a><br/> <a href="https://jira.whamcloud.com/browse/LU-762" title="Hyperion - mdtest failure" class="issue-link" data-issue-key="LU-762"><del>LU-762</del></a> ldiskfs: don't drop directory nlink to 0 (Revision 51f76e02e2a04070ff2125285642430f703962ac)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=51f76e02e2a04070ff2125285642430f703962ac" class="external-link" target="_blank" rel="nofollow noopener">51f76e02e2a04070ff2125285642430f703962ac</a><br/> Files : </p> <ul> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.series</li> <li>ldiskfs/kernel_patches/patches/ext4-nlink-2.6-rhel5.patch</li> <li>ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel5-ext4.series</li> </ul> <p>Close first. Please feel free to reopen.</p> Development Rank 1|hzv84v: Rank (Obsolete) 4852 Severity