https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1895 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/da5b8f26-f92d-11e1-a1b8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/da5b8f26-f92d-11e1-a1b8-52540035b04c</a>.</p> <p>The sub-test test_5 failed with the following error:</p> <blockquote> <p>test_5 failed with 1</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == mmp test 5: mount during unmount of the first filesystem ========================================== 07:54:47 (1347029687) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-MDS\/P1 on mds1... Lustre: DEBUG MARKER: Mounting /dev/lvm-MDS/P1 on mds1... Lustre: DEBUG MARKER: /usr/sbin/lctl mark Unmounting \/dev\/lvm-MDS\/P1 on mds1... Lustre: DEBUG MARKER: Unmounting /dev/lvm-MDS/P1 on mds1... Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-MDS\/P1 on mds1failover... Lustre: DEBUG MARKER: Mounting /dev/lvm-MDS/P1 on mds1failover... Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-OSS\/P1 on ost1... Lustre: DEBUG MARKER: Mounting /dev/lvm-OSS/P1 on ost1... Lustre: DEBUG MARKER: /usr/sbin/lctl mark Unmounting \/dev\/lvm-OSS\/P1 on ost1... Lustre: DEBUG MARKER: Unmounting /dev/lvm-OSS/P1 on ost1... Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-OSS\/P1 on ost1failover... Lustre: DEBUG MARKER: Mounting /dev/lvm-OSS/P1 on ost1failover... Lustre: DEBUG MARKER: /usr/sbin/lctl mark mmp test_5: @@@@@@ FAIL: mount during unmount of the first filesystem should fail </pre> </div></div> LU-1895

Failover Test failure on test suite mmp, subtest test_5

Bug Major Resolved Fixed Andreas Dilger Maloo yuc2 Tue, 11 Sep 2012 14:24:53 +0000 Tue, 11 Sep 2018 20:44:56 +0000 Wed, 15 Aug 2018 21:39:52 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.4 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.5 Lustre 2.12.0 Lustre 2.10.6 0 11 <p>Lustre Tag: v2_3_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/32" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/32</a><br/> Distro/Arch: RHEL6.3/x86_64(server), RHEL5.8/x86_64(client)<br/> Test Group: failover</p> <p>The issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/37aa5cec-146e-11e2-af8d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/37aa5cec-146e-11e2-af8d-52540035b04c</a></p> <p>Lustre Tag: v2_1_4_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/159/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/159/</a><br/> Distro/Arch: RHEL6.3/x86_64</p> <p>This issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/1ce16246-4bb6-11e2-aa80-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1ce16246-4bb6-11e2-aa80-52540035b04c</a></p> <p>mmp passed in the latest tag 2.3.62 failover testing:</p> <p><a href="https://maloo.whamcloud.com/test_sets/9e700da0-8b3d-11e2-965f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9e700da0-8b3d-11e2-965f-52540035b04c</a></p> <p>Lustre Branch: master<br/> Distro/Arch: RHEL6.4/x86_64<br/> Test Group: failover</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1486" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1486</a><br/> <a href="https://maloo.whamcloud.com/test_sets/9192eca4-bb19-11e2-8824-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9192eca4-bb19-11e2-8824-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1481" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1481</a><br/> <a href="https://maloo.whamcloud.com/test_sets/e7c55d7c-b8f9-11e2-891d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e7c55d7c-b8f9-11e2-891d-52540035b04c</a></p> <p>Per analysis from Andreas, reducing from blocker.</p> <p>Yu Jian,<br/> it looks like this is potentially a race in the test case? We don't run mmp.sh test_5 very often - only in "failover" mode. I was working on <a href="http://review.whamcloud.com/3715" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3715</a> to try and get these tests to run in more cases, but I haven't completed it. In theory, we should be able to run the MMP tests all the time, since the VM nodes are all on the same hosts.</p> <p>We're reducing this from a blocker, since I think the problem is in the test case and not in the code. Please change it back if you disagree.</p> <p>Lustre Tag: v2_4_0_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1501/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1501/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Test Group: failover</p> <p>The issue occurred again:<br/> <a href="https://maloo.whamcloud.com/test_sets/b0bf3f88-c00f-11e2-8398-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b0bf3f88-c00f-11e2-8398-52540035b04c</a></p> <p>Unmounting /dev/lvm-OSS/P1 on ost1 (client-31vm4):</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>11:55:51:Lustre: DEBUG MARKER: Unmounting /dev/lvm-OSS/P1 on ost1...<br/> 11:55:52:Lustre: DEBUG MARKER: grep -c /mnt/ost1' ' /proc/mounts<br/> 11:55:52:Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-OSS\/P1 on ost1failover...<br/> 11:55:52:Lustre: DEBUG MARKER: Mounting /dev/lvm-OSS/P1 on ost1failover...<br/> 11:55:52:Lustre: DEBUG MARKER: umount -d /mnt/ost1<br/> 11:55:52:Lustre: Failing over lustre-OST0000<br/> <font color="red">11:55:52</font>:Lustre: server umount lustre-OST0000 complete</p> </div></div> <p>Mounting /dev/lvm-OSS/P1 on ost1failover (client-31vm8):</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>11:55:52:Mounting /dev/lvm-OSS/P1 on ost1failover...<br/> 11:55:52:CMD: client-31vm4 grep -c /mnt/ost1' ' /proc/mounts<br/> 11:55:52:Stopping /mnt/ost1 (opts<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> on client-31vm4<br/> 11:55:52:CMD: client-31vm4 umount -d /mnt/ost1<br/> 11:55:52:CMD: client-31vm8 mkdir -p /mnt/ost1failover<br/> 11:55:53:CMD: client-31vm8 test -b /dev/lvm-OSS/P1<br/> 11:55:53:CMD: client-31vm4 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST '<br/> 11:55:53:Starting ost1failover: /dev/lvm-OSS/P1 /mnt/ost1failover<br/> <font color="red">11:55:53</font>:CMD: client-31vm8 mkdir -p /mnt/ost1failover; mount -t lustre /dev/lvm-OSS/P1 /mnt/ost1failover<br/> 11:55:53:CMD: client-31vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin::/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super\" \"all -lnet -lnd -pinger\" 48 <br/> 11:55:54:CMD: client-31vm8 e2label /dev/lvm-OSS/P1 2&gt;/dev/null<br/> 11:55:54:Started lustre-OST0000</p> </div></div> <p>This is a test script issue.</p> <p>Lustre Tag: v2_4_1_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Testgroup: failover</p> <p>mmp test 5 failed again:<br/> <a href="https://maloo.whamcloud.com/test_sets/21323846-194e-11e3-bb73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/21323846-194e-11e3-bb73-52540035b04c</a></p> <p>also seen in tag-2.3.54 testing:</p> <p>server and client: lustre-master build # 1837 RHEL6 ldiskfs</p> <p><a href="https://maloo.whamcloud.com/test_sets/a6b7e1f4-8411-11e3-a854-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a6b7e1f4-8411-11e3-a854-52540035b04c</a></p> <p>another instance in 2.6<br/> <a href="https://maloo.whamcloud.com/test_sets/e8249cd2-a418-11e3-8c6a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e8249cd2-a418-11e3-8c6a-52540035b04c</a></p> <p>hit this in 2.7.0-rc2 testing</p> <p><a href="https://testing.hpdd.intel.com/test_sets/0ee08f32-c157-11e4-9d86-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0ee08f32-c157-11e4-9d86-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL6.7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/47839e28-bc93-11e5-8f65-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/47839e28-bc93-11e5-8f65-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL6.7 Server/SLES11 SP3 Clients<br/> <a href="https://testing.hpdd.intel.com/test_sets/96492436-ba4c-11e5-9a07-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/96492436-ba4c-11e5-9a07-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/3506468a-bc00-11e5-a592-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3506468a-bc00-11e5-a592-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL7 Server/SLES11 SP3 Client<br/> build# 3303<br/> <a href="https://testing.hpdd.intel.com/test_sets/900a5b64-bb2b-11e5-b3d5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/900a5b64-bb2b-11e5-b3d5-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL6.7 Server/Client, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/7c5e8006-cb2d-11e5-b3e8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/7c5e8006-cb2d-11e5-b3e8-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/Client, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/SLES11 SP3 Client, tag 2.7.66, master build 3316<br/> <a href="https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2</a></p> <p>Another instance found on b2_8 for failover testing , build# 6.<br/> <a href="https://testing.hpdd.intel.com/test_sessions/0aed3028-da39-11e5-a8a6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0aed3028-da39-11e5-a8a6-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/eaf85780-d65e-11e5-afe8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eaf85780-d65e-11e5-afe8-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2</a></p> <p>2.9.57, b3575:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/1f3eb5f9-bb00-4ea9-be10-4318eabbff04" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/1f3eb5f9-bb00-4ea9-be10-4318eabbff04</a></p> <p>2.10.1 b26:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/2ecbd0a0-6511-479f-b014-0222fd7009b1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2ecbd0a0-6511-479f-b014-0222fd7009b1</a></p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32355" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32355</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1895" title="Failover Test failure on test suite mmp, subtest test_5" class="issue-link" data-issue-key="LU-1895"><del>LU-1895</del></a> tests: don't fail mmp test_5 due to race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a0d9f2db4987fcf44176f60a6d060d4bba788b13</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33007" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33007</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1895" title="Failover Test failure on test suite mmp, subtest test_5" class="issue-link" data-issue-key="LU-1895"><del>LU-1895</del></a> tests: don't fail mmp test_5 due to race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 63b5d72a43765d8f54fba884f752d658e9a0daea</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32355/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32355/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1895" title="Failover Test failure on test suite mmp, subtest test_5" class="issue-link" data-issue-key="LU-1895"><del>LU-1895</del></a> tests: don't fail mmp test_5 due to race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7e7a4c9402d464d6b3e67a29a1b5c7e02acc4836</p> <p>Landed for 2.12</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33007/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33007/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1895" title="Failover Test failure on test suite mmp, subtest test_5" class="issue-link" data-issue-key="LU-1895"><del>LU-1895</del></a> tests: don't fail mmp test_5 due to race<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 9632e97b907fd55e0ec1f652fa936ecd24720e25</p> Development Rank 1|hzv3lj: Rank (Obsolete) 4082 Severity