https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13160 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/f4707d44-3ba7-11ea-bb75-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f4707d44-3ba7-11ea-bb75-52540065bddc</a></p> <p>test_70 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================= 06:37:48 (1579415868) CMD: trevis-12vm7 mktemp --tmpdir=/tmp -d sanity-hsm.test_70.XXXX CMD: trevis-12vm7 mkfifo -m 0644 /tmp/sanity-hsm.test_70.r3C7/fifo CMD: trevis-12vm7 cat /tmp/sanity-hsm.test_70.r3C7/fifo &gt; /tmp/sanity-hsm.test_70.r3C7/events &amp; echo \$! &gt; /tmp/sanity-hsm.test_70.r3C7/monitor_pid </pre> </div></div> <p>Timeout occurred after 238 mins, last suite running was sanity-hsm, restarting cluster to continue tests</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-hsm test_70 - Timeout occurred after 238 mins, last suite running was sanity-hsm, restarting cluster to continue tests</p> RHEL 8.1 client + RHEL 7.7 server LU-13160

sanity-hsm test 70 timeout

Bug Minor Resolved Fixed Dongyang Li Maloo rhel8 Tue, 21 Jan 2020 07:16:15 +0000 Wed, 20 May 2020 23:13:38 +0000 Wed, 20 May 2020 23:13:38 +0000 Lustre 2.13.0 Lustre 2.12.4 Lustre 2.14.0 Lustre 2.12.5 0 6 <p>Yingjin</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>The failure also occurred on Lustre b2_12 branch with RHEL 8.1 client:<br/> <a href="https://testing.whamcloud.com/test_sets/59bb2c92-3ea7-11ea-9543-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/59bb2c92-3ea7-11ea-9543-52540065bddc</a></p> <p>Dongyang</p> <p>Could you please look into why this regression has appeared using RHEL 8.1 rather than RHEL 8.0 clients?</p> <p>Peter</p> <p>from the both test sessions, seems the process was stuck after the last cat command started on the second client. the stack log from the session suggests the cat process was indeed started:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [22875.131497] cat S 0 1279 1 0x00000080 [22875.132478] Call Trace: [22875.132975] ? __schedule+0x253/0x830 [22875.133660] schedule+0x28/0x70 [22875.134253] pipe_wait+0x6c/0xb0 [22875.134880] ? finish_wait+0x80/0x80 [22875.135566] wait_for_partner+0x19/0x50 [22875.136269] fifo_open+0x27b/0x2b0 [22875.136911] ? pipe_release+0xa0/0xa0 [22875.137597] do_dentry_open+0x132/0x330 [22875.138310] path_openat+0x573/0x14d0 [22875.139008] do_filp_open+0x93/0x100 [22875.139681] ? __check_object_size+0xa3/0x181 [22875.140466] do_sys_open+0x184/0x220 [22875.141149] do_syscall_64+0x5b/0x1b0 [22875.141843] entry_SYSCALL_64_after_hwframe+0x65/0xca </pre> </div></div> <p>which is part of the copytool_monitor_setup,</p> <p>we should proceed to </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> copytool setup --event-fifo <span class="code-quote">"$HSMTOOL_MONITOR_DIR/fifo"</span> </pre> </div></div> <p>and should be able to see</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CMD: trevis-16vm6 mkdir -p /tmp/arc1/sanity-hsm.test_70/ </pre> </div></div> <p>as the next command on the second client.</p> <p>but I couldn't see anything stopping us from doing that.</p> <p>Also tried to reproduce this with the same setup, no luck yet.</p> <p>Need some help here.</p> <p>Hi Dongyang,</p> <blockquote><p>Also tried to reproduce this with the same setup, no luck yet.</p></blockquote> <p>I can reproduce this with two clients. I found the test passed with one client. Did you try to use two clients to reproduce the issue?</p> <p>Yes I was using 2 clients.</p> <p>Are you reproducing it with the lab vms? if so can you create a setup and make sure the test case fails, then I can get into the vm and have a look?</p> <p>Thanks a lot</p> <p>Sure, Dongyang.</p> <p>Here are the test nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2 Clients: onyx-22vm3 (local), onyx-22vm5 (remote) 1 MGS/MDS: onyx-22vm1 (1 MDT) 1 OSS: onyx-22vm4 (2 OSTs) </pre> </div></div> <p>Now sanity-hsm test 70 is hanging on onyx-22vm3:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================= 06:18:09 (1581574689) CMD: onyx-22vm5 mktemp --tmpdir=/tmp -d sanity-hsm.test_70.XXXX CMD: onyx-22vm5 mkfifo -m 0644 /tmp/sanity-hsm.test_70.Whg8/fifo CMD: onyx-22vm5 cat /tmp/sanity-hsm.test_70.Whg8/fifo &gt; /tmp/sanity-hsm.test_70.Whg8/events &amp; echo \$! &gt; /tmp/sanity-hsm.test_70.Whg8/monitor_pid </pre> </div></div> <p>Many thanks Jian,</p> <p>Looks like the session on onyx-22vm3 has already finished, I was trying to start a new session with something like --only 70 but I noticed the cfg/local.sh was not setup.</p> <p>Can I just use auster under /usr/lib64/lustre/tests on 22vm3? if not can you provide a cmd for it?</p> <p>Sure, Dongyang.<br/> I usually directly specify variable values to run auster. Here is the command I ran on onyx-22vm3 under /usr/lib64/lustre/tests:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm3 tests]# PDSH="pdsh -t 120 -S -R ssh -w" NAME=ncli RCLIENTS=onyx-22vm5 mds_HOST=onyx-22vm1 MDSDEV=/dev/vda5 MDSSIZE=2097152 ost_HOST=onyx-22vm4 OSTCOUNT=2 OSTSIZE=2097152 OSTDEV1=/dev/vda5 OSTDEV2=/dev/vda6 SHARED_DIRECTORY="/home/jianyu/test_logs" VERBOSE=true bash auster -d /home/jianyu/test_logs -r -s -v -k sanity-hsm --only 70 </pre> </div></div> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37595" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37595</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13160" title="sanity-hsm test 70 timeout" class="issue-link" data-issue-key="LU-13160"><del>LU-13160</del></a> tests: fix sanity-hsm monitor setup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 093d9f2fe4d80060db46475bf32d4b985edebf97</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37595/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37595/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13160" title="sanity-hsm test 70 timeout" class="issue-link" data-issue-key="LU-13160"><del>LU-13160</del></a> tests: fix sanity-hsm monitor setup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6724d8ca58e9b8474a180b013a4723cbdd8900d9</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37773" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37773</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13160" title="sanity-hsm test 70 timeout" class="issue-link" data-issue-key="LU-13160"><del>LU-13160</del></a> tests: fix sanity-hsm monitor setup<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 2f639102be1ded4504027a735affa041dde3552a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37773/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37773/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13160" title="sanity-hsm test 70 timeout" class="issue-link" data-issue-key="LU-13160"><del>LU-13160</del></a> tests: fix sanity-hsm monitor setup<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: c0a877ab3b049266042299a438d8d010ce3ce605</p> <p>Is this complete?</p> <p>Yes, I'm closing the ticket.</p> Development Rank 1|i00sa7: Rank (Obsolete) 9223372036854775807 Severity