https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17154 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/568e0c21-9347-476a-beac-081e9b2ee112" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/568e0c21-9347-476a-beac-081e9b2ee112</a></p> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-master/4468" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4468</a> - 5.14.0-284.25.1.el9_2.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-master/4468" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4468</a> - 5.14.0-284.25.1_lustre.el9.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>parallel-scale-nfsv4 hangs on:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Stopping client trevis-27vm4 /mnt/lustre (opts:-f) CMD: trevis-27vm4 lsof -t /mnt/lustre pdsh@trevis-27vm1: trevis-27vm4: ssh exited with exit code 1 CMD: trevis-27vm4 umount -f /mnt/lustre 2&gt;&amp;1 </pre> </div></div> <p>Console long on trevis-27vm4:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[70712.060132] Lustre: DEBUG MARKER: umount -f /mnt/lustre 2&gt;&amp;1 [70712.213680] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [70712.215066] LustreError: 2067684:0:(file.c:245:ll_close_inode_openhandle()) lustre-clilmv-ffffa03d96d7f000: inode [0x200000bd3:0x2c31:0x0] mdc close failed: rc = -108 [70712.243116] Lustre: 1411383:0:(llite_lib.c:3965:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.38.143@tcp:/lustre/fid: [0x28000040a:0x3699:0x0]/ may get corrupted (rc -108) [70712.243167] Lustre: 1411382:0:(llite_lib.c:3965:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.38.143@tcp:/lustre/fid: [0x2c000040a:0x3318:0x0]/ may get corrupted (rc -108) &lt;~snip~&gt; [70742.217783] Lustre: lustre-MDT0000: haven't heard from client 0e545e12-9ad6-4857-a78b-e65f011477b4 (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 00000000320f809c, cur 1695838270 expire 1695838240 last 1695838239 [70745.262062] Lustre: lustre-MDT0002: haven't heard from client 0e545e12-9ad6-4857-a78b-e65f011477b4 (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000887f97a0, cur 1695838273 expire 1695838243 last 1695838239 </pre> </div></div> LU-17154

parallel-scale-nfsv4: hangs on umount after racer_on_nfs

Bug Minor Open Unresolved Alex Deiter Maloo Thu, 28 Sep 2023 04:26:44 +0000 Thu, 25 Jan 2024 08:36:57 +0000 Lustre 2.16.0 0 4 <p>It looks like parallel-scale-nfsv4 is failing to unmount cleanly when racer is run, and unmounted properly when racer_on_nfs is skipped. I haven't checked why racer is skipped, but I wonder if it should always be skipped? Otherwise, it makes all of the NFS testing unreliable? </p> <p>It does appear that there are some cases where racer_on_nfs is run AND the test unmounts properly, but unfortunately it is difficult to search for this in Maloo easily. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=Deiter" class="user-hover" rel="Deiter">Deiter</a>, could you please submit a patch to master and b_es6_0 to add racer_on_nfs to the always_except list. It looks like the <tt>parallel&#45;scale&#45;nfs&#45;<b>v4</b>.sh</tt> script is incorrectly checking:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> export ALWAYS_EXCEPT=<span class="code-quote">"$PARALLEL_SCALE_NFSV3_EXCEPT "</span> </pre> </div></div> <p>so this should also be updated to check <tt>PARALLEL_SCALE_NFSV4_EXCEPT</tt>. It may be that this is why racer_on_nfs is sometimes being skipped when running <tt>parallel&#45;scale&#45;nfsv4</tt> - when it is run after <tt>paralle\l-scale&#45;nfsv3</tt>?</p> <p>It would also be useful to see if the passing cases (without unmount timeout) are only for interop (I saw one case with ddn31 or similar, not sure of others). Checking this, and going back through the test history to see if there was a time this was passing regularly would help isolate if there was a patch landed that caused this problem. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a>, can you please re-add NFS testing to your patches, but skip racer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: trivial Test-Parameters: testlist=env=PARALLEL_SCALE_NFSV3_EXCEPT=racer_on_nfs </pre> </div></div> <p>This should hopefully now pass. </p> <p>Sure, <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>. I found ALWAYS_EXCEPT was not defined in parallel-scale-nfsv4.sh, so while updating patch <a href="https://review.whamcloud.com/52533" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52533</a>, I added it and also added racer_on_nfs test into the always_except list.</p> <p>You are right. I guess I was mistakenly looking at the nfsv3.sh file, and there is nothing in the nfsv4.sh file that allows setting <tt>ALWAYS_EXCEPT</tt>.</p> Related LU-17152 LU-10851 Development Rank 1|i03wzz: Rank (Obsolete) 9223372036854775807 Severity