https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11256 Lustre <p>replay-vbr test_7f fails on mounting an MDS. It’s not clear when this test started failing with this error, but it looks like this test didn’t fail on MDS mount for two and a half months and started failing again on July 14, 2018. All failures of this type since March of 2018 are listed below.</p> <p>Looking at the failure at <a href="https://testing.whamcloud.com/test_sets/5b253cd8-878f-11e8-9028-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5b253cd8-878f-11e8-9028-52540065bddc</a>, in the test_log, the only sign of trouble is when we try and mount the failover MDS </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Failing mds1 on trevis-4vm8 + pm -h powerman --off trevis-4vm8 Command completed successfully reboot facets: mds1 + pm -h powerman --on trevis-4vm8 Command completed successfully Failover mds1 to trevis-4vm7 12:30:33 (1531571433) waiting for trevis-4vm7 network 900 secs ... 12:30:33 (1531571433) network interface is UP CMD: trevis-4vm7 hostname mount facets: mds1 CMD: trevis-4vm7 test -b /dev/lvm-Role_MDS/P1 CMD: trevis-4vm7 e2label /dev/lvm-Role_MDS/P1 trevis-4vm7: e2label: No such file or directory while trying to open /dev/lvm-Role_MDS/P1 trevis-4vm7: Couldn't find valid filesystem superblock. Starting mds1: -o loop /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 CMD: trevis-4vm7 mkdir -p /mnt/lustre-mds1; mount -t lustre -o loop /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 trevis-4vm7: mount: /dev/lvm-Role_MDS/P1: failed to setup loop device: No such file or directory Start of /dev/lvm-Role_MDS/P1 on mds1 failed 32 replay-vbr test_7f: @@@@@@ FAIL: Restart of mds1 failed! </pre> </div></div> <p>In all the following cases, test 7g hangs when test 7f fails in this way.</p> <p>2018-08-15 2.10.5 RC2 – fails in “test_7f.5 last”<br/> <a href="https://testing.whamcloud.com/test_sets/a75d306e-a081-11e8-8ee3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a75d306e-a081-11e8-8ee3-52540065bddc</a><br/> 2018-08-02 2.10.4.14 – fails in “test_7f.5 last”<br/> <a href="https://testing.whamcloud.com/test_sets/7405ad54-9645-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7405ad54-9645-11e8-a9f7-52540065bddc</a><br/> 2018-07-14 2.10.4.8 - fails in “test_7f.1 last”<br/> <a href="https://testing.whamcloud.com/test_sets/5b253cd8-878f-11e8-9028-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5b253cd8-878f-11e8-9028-52540065bddc</a><br/> 2018-04-12 2.11.50.51 - fails in “test_7f.4 last”<br/> <a href="https://testing.whamcloud.com/test_sets/37bad538-3e69-11e8-b45c-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/37bad538-3e69-11e8-b45c-52540065bddc</a><br/> 2018-03-03 2.10.3.35 - fails in “test_7f.4 last”<br/> <a href="https://testing.whamcloud.com/test_sets/f33a4326-1f0f-11e8-a6ca-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f33a4326-1f0f-11e8-a6ca-52540065bddc</a></p> <p>In the following test session, replay-vbr test 7e fails in the way described above and test 7f hangs<br/> 2018-07-15 2.10.4.8 - fails in “test_7e.5 last”<br/> <a href="https://testing.whamcloud.com/test_sets/0ca6ca46-87fc-11e8-b376-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0ca6ca46-87fc-11e8-b376-52540065bddc</a><br/> 2018-03-14 2.10.59 - fails in “test_7e.5 last”<br/> <a href="https://testing.whamcloud.com/test_sets/d26f60b0-2809-11e8-b6a0-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d26f60b0-2809-11e8-b6a0-52540065bddc</a></p> LU-11256

replay-vbr test 7f is failing with 'Restart of mds1 failed!'

Bug Minor Open Unresolved WC Triage James Nunez Wed, 15 Aug 2018 20:06:40 +0000 Fri, 30 Nov 2018 18:49:34 +0000 Lustre 2.12.0 Lustre 2.10.4 Lustre 2.10.5 0 2 <p>It looks like there is a check in <tt>mount_facet()</tt> that is checking if the "device" is a block device, and if not then add "<tt>-o loop</tt>" to the mount options.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> mount_facet() { <span class="code-keyword">if</span> [ $(facet_fstype $facet) == ldiskfs ] &amp;&amp; ! do_facet $facet test -b ${!dev}; then opts=$(csa_add <span class="code-quote">"$opts"</span> -o loop) fi <span class="code-keyword">case</span> $fstype in ldiskfs) devicelabel=$(do_facet ${facet} <span class="code-quote">"$E2LABEL ${!dev}"</span>);; esac echo <span class="code-quote">"Starting ${facet}: $opts ${!dev} $mntpt"</span> <span class="code-keyword">if</span> [ $RC -ne 0 ]; then echo <span class="code-quote">"Start of ${!dev} on ${facet} failed ${RC}"</span> <span class="code-keyword">return</span> $RC fi </pre> </div></div> <p>To further debug this issue, it would make sense to add some additional debugging into the "<tt>test -b</tt>" failure case to determine if the device even exists:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> [ $(facet_fstype $facet) == ldiskfs ] &amp;&amp; ! do_facet $facet test -b ${!dev}; then <span class="code-keyword">if</span> ! do_facet $facet test -e ${!dev}; then do_facet $facet <span class="code-quote">"ls -lR $(dirname ${!dev})"</span> error <span class="code-quote">"$facet: device ${!dev} does not exist"</span> fi opts=$(csa_add <span class="code-quote">"$opts"</span> -o loop) fi </pre> </div></div> <p>Similar failure on recovery-scale-mds in test failover_mds at <a href="https://testing.whamcloud.com/test_sets/4e5eb398-a24d-11e8-a5f2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4e5eb398-a24d-11e8-a5f2-52540065bddc</a></p> Related LU-10519 LU-10708 Development Rank 1|i000t3: Rank (Obsolete) 9223372036854775807 Severity