https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9487 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/adcc6618-5344-4cda-b96f-04bee182c49b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/adcc6618-5344-4cda-b96f-04bee182c49b</a></p> <p>New for tag 2.9.57 - not seen in past 6 months (Nov '16 - May '17).</p> <p>The same failure message is seen with 7 subsequent subtests.</p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-45vm3 mkdir -p /mnt/lustre-mds1failover; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1failover trevis-45vm3 (MDS1-1): mount.lustre: according to /etc/mtab /dev/mapper/lvm--Role_MDS-P1 is already mounted on /mnt/lustre-mds1failover Start of /dev/lvm-Role_MDS/P1 on mds1failover failed 17 trevis-45vm7 (MDS1-2): mount.lustre: mount /dev/mapper/lvm--Role_MDS-P1 at /mnt/lustre-mds1 failed: Invalid argument trevis-45vm7: This may have multiple causes. trevis-45vm7: Are the mount options correct? trevis-45vm7: Check the syslog for more info. Start of /dev/lvm-Role_MDS/P1 on mds1 failed 22 mmp test_2: @@@@@@ FAIL: mount failure on failover pair mds1,mds1failover Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4927:error_noexit() = /usr/lib64/lustre/tests/mmp.sh:284:mount_after_interval_sub() = /usr/lib64/lustre/tests/mmp.sh:296:mount_after_interval() = /usr/lib64/lustre/tests/mmp.sh:446:test_2() = /usr/lib64/lustre/tests/test-framework.sh:5215:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5254:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5101:run_test() = /usr/lib64/lustre/tests/mmp.sh:449:main() Dumping lctl log to /test_logs/2017-05-08/lustre-master-el7-x86_64--failover--1_17_1__3575__-70321399293120-051547/mmp.test_2.*.1494260419.log CMD: trevis-45vm1.trevis.hpdd.intel.com,trevis-45vm5,trevis-45vm6,trevis-45vm7,trevis-45vm8 /usr/sbin/lctl dk &gt; /test_logs/2017-05-08/lustre-master-el7-x86_64--failover--1_17_1__3575__-70321399293120-051547/mmp.test_2.debug_log.\$(hostname -s).1494260419.log; dmesg &gt; /test_logs/2017-05-08/lustre-master-el7-x86_64--failover--1_17_1__3575__-70321399293120-051547/mmp.test_2.dmesg.\$(hostname -s).1494260419.log Resetting fail_loc on all nodes...CMD: trevis-45vm1.trevis.hpdd.intel.com,trevis-45vm5,trevis-45vm6,trevis-45vm7,trevis-45vm8 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. mmp test_2: @@@@@@ FAIL: test_2 failed with 22 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4939:error() = /usr/lib64/lustre/tests/test-framework.sh:5215:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5254:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5101:run_test() = /usr/lib64/lustre/tests/mmp.sh:449:main() </pre> </div></div> trevis-45, failover<br/> &nbsp;&nbsp;EL7, master branch, v2.9.57, b3575, ldiskfs<br/> LU-9487

mmp test_2: test_2 failed with 22

Bug Major Resolved Fixed Alex Zhuravlev James Casper Wed, 10 May 2017 15:37:04 +0000 Sat, 3 Jun 2017 04:30:50 +0000 Sat, 3 Jun 2017 04:30:50 +0000 Lustre 2.10.0 Lustre 2.10.0 0 6 <p>The test set that follows mmp (replay-single) also fails with the same failure message. replay-single is the last test set in the failover group.</p> <p>It looks like the MDS didn't clean up properly at the end of the previous test? This needs to be investigated more closely.</p> <p>A clarification on this statement: "New for tag 2.9.57 - not seen in past 6 months (Nov '16 - May '17)."</p> <p>This has not been seen in tag testing since November 10, 2016 (master branch, tags 51-56). There are 55 occurrences outside of tag testing (all in failover group testing).</p> <p>Alex is looking into this one</p> <p>the log looks a bit strange..<br/> [ 307.590154] Lustre: DEBUG MARKER: ----<del>============= acceptance-small: mmp ============</del>---- Mon May 8 16:18:34 UTC 2017<br/> [ 307.956938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration -o sec ================================================================ 16:18:34 &#40;1494260314&#41;<br/> [ 308.111498] Lustre: DEBUG MARKER: == mmp test complete, duration -o sec ================================================================ 16:18:34 (1494260314)<br/> [ 336.896363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 1: two mounts at the same time =========================================================== 16:19:03 &#40;1494260343&#41;</p> <p>== mmp test 1: two mounts at the same time =========================================================== 16:19:03 (1494260343)<br/> Starting mds1failover: /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1failover<br/> Starting mds1: /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1<br/> Started lustre-MDT0000<br/> Start of /dev/lvm-Role_MDS/P1 on mds1 failed 22<br/> Starting ost1: /dev/lvm-Role_OSS/P1 /mnt/lustre-ost1<br/> Starting ost1failover: /dev/lvm-Role_OSS/P1 /mnt/lustre-ost1failover<br/> Started lustre-OST0000<br/> Start of /dev/lvm-Role_OSS/P1 on ost1failover failed 22<br/> Stopping /mnt/lustre-ost1 (opts<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> on trevis-45vm8<br/> PASS 1 (52s)</p> <p>for whatever reason MDS wasn't stopped by test 1, though I see in the scripts this:<br/> stop $mds_facet $opts || return ${PIPESTATUS<span class="error">&#91;0&#93;</span>}<br/> stop $oss_facet $opts || return ${PIPESTATUS<span class="error">&#91;0&#93;</span>}</p> <p>going to add a debugging</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27192" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27192</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9487" title="mmp test_2: test_2 failed with 22" class="issue-link" data-issue-key="LU-9487"><del>LU-9487</del></a>: debugging patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7881652b82308ef5193efa2cb85292ea540e1723</p> <p>I think the root cause is that test #2 expects the primary MDS to be mounted:<br/> stop_services primary || return ${PIPESTATUS<span class="error">&#91;0&#93;</span>}<br/> but actually the mount process in mount_after_interval_sub() seem to be racy by desing:<br/> start $facet $device $opts &amp;<br/> ...<br/> start $failover_facet $device $opts</p> <p>the test was developed by Jian Yu, it'd helpful to get any hints on this.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27253" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27253</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9487" title="mmp test_2: test_2 failed with 22" class="issue-link" data-issue-key="LU-9487"><del>LU-9487</del></a> tests: stop primary and failover services in mmp test 1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0f5520c5413e32caa55275c4dc7310e1a03a60a5</p> <p>Yes, Alex. In mmp.sh test_1(), it's likely the failover service is started before the primary one, I just pushed a patch to check and stop the failover service. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27253/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27253/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9487" title="mmp test_2: test_2 failed with 22" class="issue-link" data-issue-key="LU-9487"><del>LU-9487</del></a> tests: stop primary and failover services in mmp test 1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a23bb09a3bc447e4286218416d1dd5d096724bfc</p> <p>Landed for 2.10</p> Development Rank 1|hzzcdz: Rank (Obsolete) 9223372036854775807 Severity