https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12171 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/43fba6f6-5a10-11e9-92fe-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/43fba6f6-5a10-11e9-92fe-52540065bddc</a></p> <p>test_133g failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 161 mins, last suite running was sanity, restarting cluster to continue tests </pre> </div></div> <p>trevis-14vm9 hosting MDT0000 becomes unreachable, and this situation cannot be recovered. From the MDS standpoint, client was evicted, but the client did not manage to reconnect.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_133g - Timeout occurred after 161 mins, last suite running was sanity, restarting cluster to continue tests</p> LU-12171

sanity test_133g: Timeout occurred after 161 mins

Bug Minor Resolved Duplicate WC Triage Maloo Mon, 8 Apr 2019 15:45:47 +0000 Fri, 19 Apr 2019 18:11:15 +0000 Thu, 18 Apr 2019 22:37:32 +0000 0 4 <p>another instance:</p> <p><a href="https://testing.whamcloud.com/test_sessions/45917329-c33d-4929-9393-69b342c5f0eb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/45917329-c33d-4929-9393-69b342c5f0eb</a></p> <p><a href="https://testing.whamcloud.com/test_sessions/d70e90ee-cbfd-426e-b0b7-e82cad8bcfd4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/d70e90ee-cbfd-426e-b0b7-e82cad8bcfd4</a></p> <p>So I took a look here...</p> <p>mds1 is getting failed over in all of the failure cases.</p> <p>In the cleanup for this, we check if mds1 is active on the expected server, and if it's not, we fail it.  That's fine...</p> <p>In the timeout cases, mds1 is not showing as active, and so it's getting failed over - but it's getting failed over to the same VM it started from:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cln..Failing mds1 on onyx-43vm9 CMD: onyx-43vm9 grep -c /mnt/lustre-mds1' ' /proc/mounts || true Stopping /mnt/lustre-mds1 (opts:) on onyx-43vm9 CMD: onyx-43vm9 umount -d /mnt/lustre-mds1 CMD: onyx-43vm9 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' || true CMD: onyx-43vm9 ! zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 reboot facets: mds1 Failover mds1 to onyx-43vm9 13:57:29 (1554731849) waiting for trevis-14vm9 network 900 secs ... 13:57:29 (1554731849) network interface is UP CMD: trevis-14vm9 hostname mount facets: mds1</pre> </div></div> <p>This is quite odd.  I can't see why this would happen...  But it seems reasonable that self-failover like this, which isn't intended, might confuse something.  (Can't figure out what yet.)</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jamesanunez" class="user-hover" rel="jamesanunez">jamesanunez</a>:</p> <p>This started on April 8th, and is limited to DNE testing.  Given what we're seeing elsewhere, I'd lay money this is <b>also</b> fallout from <a href="https://jira.whamcloud.com/browse/LU-11636" title="t-f test_mkdir() does not support interop with non DNEII servers" class="issue-link" data-issue-key="LU-11636"><del>LU-11636</del></a>.</p> <p>Dupe of <a href="https://jira.whamcloud.com/browse/LU-12175" title="sanity test 208 fails with &#39;lease broken over recovery&#39;" class="issue-link" data-issue-key="LU-12175">LU-12175</a></p> Duplicate LU-12175 Related LU-12210 Development Rank 1|i00en3: Rank (Obsolete) 9223372036854775807 Severity