https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1639 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/ef641dcc-cd20-11e1-957a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ef641dcc-cd20-11e1-957a-52540035b04c</a>.</p> <p>The sub-test test_iorssf failed with the following error:</p> <blockquote> <p>test_iorssf failed with 1</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Max Write: 19.36 MiB/sec (20.30 MB/sec) Max Read: 15.61 MiB/sec (16.37 MB/sec) Run finished: Fri Jul 13 11:11:40 2012 rm: cannot remove `/mnt/lustre/d0.ior.ssf': Directory not empty parallel-scale-nfsv3 test_iorssf: @@@@@@ FAIL: test_iorssf failed with 1 Dumping lctl log to /logdir/test_logs/2012-07-12/lustre-master-el6-x86_64-ofa__713__-7f175835ae28/parallel-scale-nfsv3.test_iorssf.*.1342203102.log </pre> </div></div> LU-1639

Test failure parallel-scale-nfsv3, test_iorssf

Bug Major Open Unresolved Hongchao Zhang Maloo yuc2 Tue, 17 Jul 2012 16:14:28 +0000 Fri, 24 Jul 2020 16:37:33 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.3 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 Lustre 2.8.0 Lustre 2.10.4 1 8 <p>Hongchao, could you please look into this one?</p> <p>Hi Sarah, <br/> Is this issue reproducible, and if it is, could you please list the content in "/mnt/lustre/d0.ior.ssf" in run_ior? thanks!</p> <p>This can be reproduced on the external OFED build which I tested manually. The content in /mnt/lustre/d0.ior.ssf is iorData</p> <p>Run finished: Thu Jul 26 16:42:27 2012<br/> ++content: iorData<br/> rm: cannot remove `/mnt/lustre/d0.ior.ssf': Directory not empty<br/> parallel-scale-nfsv3 test_iorssf: @@@@@@ FAIL: test_iorssf failed with 1</p> <p>Hi Sarah,<br/> sorry, I logged in your booked node to see the logs and re-test it and it passed the first time, but the second time failed<br/> and client-4 seems to be stuck!</p> <p>I check the logs at client-4(client 1) and client-3(MDS 1), and the -ENOTEMPTY(-39) is not found.<br/> which build do you use to test? I plan to test it myself, Thanks!</p> <p>Hi Hongchao,</p> <p>Here is the build I used for testing, lustre-master-#733-RHEL6-ofa build for both server and client, you may have to manually load module mlx4_ib to make the IB work.</p> <p><a href="http://build.whamcloud.com/job/lustre-master/733/arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/733/arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/</a><br/> <a href="http://build.whamcloud.com/job/lustre-master/733/arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/733/arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/</a></p> <p>I have tested it several times without this issue, but it uses TCP for there was a problem when setup with IB (but mix4_ib was loaded).</p> <p>Hi Sarah, is the issue only be reproduced under IB?</p> <p>Hongchao,</p> <p>Yes, I only see this error in IB </p> <p>status update:</p> <p>there is some problem to reproduce the issue, and have requested some help to do it.<br/> for the current collected information, this bug could be related to NFS for there is no error found in the logs of Lustre.</p> <p>Hit this issue again in interop testing between 2.1.3 client and 2.3-RC1 server<br/> <a href="https://maloo.whamcloud.com/test_sets/dc49ce3a-079f-11e2-b8a8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc49ce3a-079f-11e2-b8a8-52540035b04c</a></p> <p>Lustre Client: v2_1_4_RC1<br/> Lustre Server: 2.1.3<br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: IB (in-kernel OFED)</p> <p><a href="https://maloo.whamcloud.com/test_sets/36e3e7b6-487f-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/36e3e7b6-487f-11e2-8cdc-52540035b04c</a></p> <p>Lustre Client: v2_1_5_RC1<br/> Lustre Server: 2.2.0<br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: IB (in-kernel OFED)</p> <p>The issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/c059a4f8-96c8-11e2-9ec7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c059a4f8-96c8-11e2-9ec7-52540035b04c</a></p> <p>Hit this issue in current tag-2.5.52 testing with DNE enabled.</p> <p><a href="https://maloo.whamcloud.com/test_sets/a3b7701c-5d26-11e3-ad71-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a3b7701c-5d26-11e3-ad71-52540035b04c</a></p> <p>client and server: lustre-master build #1791 RHEL6 ldiskfs</p> <p>test log </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Run finished: Wed Dec 4 07:08:49 2013 rm: cannot remove `/mnt/lustre/d0.ior.ssf': Directory not empty parallel-scale-nfsv3 test_iorssf: @@@@@@ FAIL: test_iorssf failed with 1 </pre> </div></div> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/63/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/63/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Network: TCP</p> <p>parallel-scale-nfsv3 test iorssf hit this failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/9a2f2c16-6022-11e3-abbc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9a2f2c16-6022-11e3-abbc-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/90200c6a-5f8a-11e3-85c5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/90200c6a-5f8a-11e3-85c5-52540035b04c</a></p> <p>It passed in another test run on the same build:<br/> <a href="https://maloo.whamcloud.com/test_sets/37bd14a2-5cdb-11e3-956b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/37bd14a2-5cdb-11e3-956b-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/67/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/67/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Network: TCP</p> <p>The same failure occurred again:<br/> <a href="https://maloo.whamcloud.com/test_sets/7d5d7bb0-6289-11e3-bf4a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d5d7bb0-6289-11e3-bf4a-52540035b04c</a></p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)<br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_4/69/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/69/</a> (2.4.2 RC1)</p> <p>The same failure occurred again:<br/> <a href="https://maloo.whamcloud.com/test_sets/808f76b6-6861-11e3-a16f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/808f76b6-6861-11e3-a16f-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2 RC2)<br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/726a1618-6a73-11e3-8e21-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/726a1618-6a73-11e3-8e21-52540035b04c</a></p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_1/220/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/220/</a> (2.1.6)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/cf401804-730d-11e3-9955-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cf401804-730d-11e3-9955-52540035b04c</a></p> <p>More instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/08c7a914-8639-11e3-8155-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/08c7a914-8639-11e3-8155-52540035b04c</a></p> <p>this should be an issue related to NFS, which doesn't send deletion requests for the child dentries before deleting the parent dentry by "rm -rf" command</p> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=770250" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=770250</a><br/> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=814052" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=814052</a></p> <p>will create a debug patch to verify whether it is the case or not.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#/c/9009/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9009/</a></p> <p>Hi YuJian, could you please test with the patch to reproduce the issue, Thanks!</p> <blockquote><p>Hi YuJian, could you please test with the patch to reproduce the issue, Thanks!</p></blockquote> <p>Hi Hongchao,</p> <p>Please add the following test parameters into commit message to reproduce the failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: fortestonly allwaysuploadlogs \ envdefinitions=SLOW=yes,ENABLE_QUOTA=yes \ testlist=parallel-scale-nfsv3,parallel-scale-nfsv3 </pre> </div></div> <p>The test name can be specified for multiple times.</p> <p>from the output, this issue can be verified to be the NFS issue,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">execve(<span class="code-quote">"/bin/rm"</span>, [<span class="code-quote">"rm"</span>, <span class="code-quote">"-rf"</span>, <span class="code-quote">"/mnt/lustre/d0.ior.ssf"</span>], mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbb6796b000 ... fcntl(4, F_GETFD) = 0 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 getdents(3, {{d_ino=288230393331613511, d_off=1, d_reclen=24, d_name=<span class="code-quote">"."</span>} {d_ino=144115188193296385, d_off=2, d_reclen=24, d_name=<span class="code-quote">".."</span>} {d_ino=288230393331613513, d_off=3, d_reclen=32, d_name=<span class="code-quote">"iorData"</span>}}, 262144) = 80 getdents(3, {{d_ino=288230393331613513, d_off=3, d_reclen=32, d_name=<span class="code-quote">"iorData"</span>}}, 262144) = 32 getdents(3, 0x7fbb6791e038, 262144) = -1 ELOOP (Too many levels of symbolic links) </pre> </div></div> <p>ELOOP is encountered during read the subdirectories.</p> <p><a href="https://maloo.whamcloud.com/test_logs/f6ddc006-88d3-11e3-b1c0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/f6ddc006-88d3-11e3-b1c0-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1)<br/> Distro/Arch: RHEL6.5/x86_64<br/> MDSCOUNT=2</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/fd7da6b0-a557-11e3-a61d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fd7da6b0-a557-11e3-a61d-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/61/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/61/</a><br/> Distro/Arch: RHEL6.5/x86_64</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/0ac3bd28-eafe-11e3-966a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0ac3bd28-eafe-11e3-966a-52540035b04c</a></p> <p>Hit this error in lustre-b2_6-RC2 testing<br/> server and client: RHEL6 ldiskfs</p> <p><a href="https://testing.hpdd.intel.com/test_sets/dd820620-0dc7-11e4-af8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dd820620-0dc7-11e4-af8b-5254006e85c2</a></p> <p>similar failure:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/dd48bbae-255e-11e5-a713-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dd48bbae-255e-11e5-a713-5254006e85c2</a></p> <p>Another instance found for interop : 2.5.5 Server/EL6.7 Client<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: master, build# 3303, RHEL 6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/16c36b0c-bb25-11e5-861c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/16c36b0c-bb25-11e5-861c-5254006e85c2</a></p> <p>Encountered same issue for tag 2.7.66 for FULL- EL7.1 Server/EL6.7 Client , master , build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/91eaee06-ca91-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/91eaee06-ca91-11e5-9609-5254006e85c2</a></p> <p>Another failure for master : Tag 2.7.66 FULL - EL7.1 Server/SLES11 SP3 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b154fcf2-ca7b-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b154fcf2-ca7b-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL7.1 Server/EL6.7 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/91eaee06-ca91-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/91eaee06-ca91-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/SLES11 SP3 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b154fcf2-ca7b-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b154fcf2-ca7b-11e5-9609-5254006e85c2</a></p> <p>2.10.3_132 &lt;-&gt; EE3</p> <p><a href="https://testing.hpdd.intel.com/test_sets/ca38c5f8-509f-11e8-abc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ca38c5f8-509f-11e8-abc3-52540065bddc</a></p> Development Rank 1|hzvdg7: Rank (Obsolete) 5785 Severity