https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9200 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>Please provide additional information about the failure here.</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/f11eec1a-f8f8-11e6-aac4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f11eec1a-f8f8-11e6-aac4-5254006e85c2</a>.</p> <p>server and client: lustre-master tag-2.9.53, el7 zfs</p> <p>there is no error log, this failure causes sanity-lfsck cannot be run, also affect the following 2 test suites.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Starting ost1: lustre-ost1/ost1 /mnt/lustre-ost1 CMD: trevis-36vm8 mkdir -p /mnt/lustre-ost1; mount -t lustre lustre-ost1/ost1 /mnt/lustre-ost1 trevis-36vm8: mount.lustre: mount lustre-ost1/ost1 at /mnt/lustre-ost1 failed: Cannot send after transport endpoint shutdown </pre> </div></div> LU-9200

sanity-lfsck: failed to mount OST

Bug Minor Open Unresolved WC Triage Maloo Thu, 9 Mar 2017 21:41:31 +0000 Wed, 3 Mar 2021 20:55:04 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.10.5 Lustre 2.10.7 0 4 <p>There isn't much in the logs. It looks like there may have been a service still running from the previous test that was not unmounted cleanly, and this was preventing the new OST mount from failing. Will see if this happens again.</p> <p>We have seen this failure several times since this ticket was opened. A recent example of this failure is at<br/> <a href="https://testing.hpdd.intel.com/test_sets/874a01ea-111f-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/874a01ea-111f-11e8-a7cd-52540065bddc</a></p> <p>In the OSS console log, we see the mount fail</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 274.852821] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1 [ 275.205895] Lustre: DEBUG MARKER: lsmod | grep zfs &gt;&amp;/dev/null || modprobe zfs; [ 275.205895] zpool list -H lustre-ost1 &gt;/dev/null 2&gt;&amp;1 || [ 275.205895] zpool import -f -o cachefile=none -o failmode=panic -d /dev/lvm-Role_OSS lustre-ost1 [ 275.688869] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-ost1/ost1 [ 276.020363] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre lustre-ost1/ost1 /mnt/lustre-ost1 [ 284.401047] LustreError: 15f-b: lustre-OST0000: cannot register this server with the MGS: rc = -108. Is the MGS running? [ 284.403459] LustreError: 9448:0:(obd_mount_server.c:1934:server_fill_super()) Unable to start targets: -108 [ 284.405669] LustreError: 9448:0:(obd_mount_server.c:1586:server_put_super()) no obd lustre-OST0000 [ 284.408071] LustreError: 9448:0:(obd_mount_server.c:132:server_deregister_mount()) lustre-OST0000 not registered [ 284.410875] Lustre: server umount lustre-OST0000 complete [ 284.412717] LustreError: 9448:0:(obd_mount.c:1583:lustre_fill_super()) Unable to mount (-108) </pre> </div></div> <p>In this case, lustre-initialization took place right before sanity-lfsck failed mounting OST0. </p> <p>There's, what looks like, another instance of this with Ubuntu clients, at <a href="https://testing.hpdd.intel.com/test_sets/49dd5b22-12b9-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/49dd5b22-12b9-11e8-a6ad-52540065bddc</a>. In this case, the console log for the OSS has an interesting message about the obd reference</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 242.176504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? </pre> </div></div> <p>AFAICS, I'm getting this very frequently. for example, <a href="https://testing.hpdd.intel.com/test_sessions/bdf41958-f133-4ab6-a190-a7d9cdfd785f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/bdf41958-f133-4ab6-a190-a7d9cdfd785f</a></p> <p>For Lustre 2.10.5 RC1, we see this issue with the OST mount cause sanity-lfsck to fail without running any tests, but we also see the next three test suites; sanityn, sanity-hsm, sanity-lsnapshot; fail do to this issue. </p> <p>An example of this is at <a href="https://testing.whamcloud.com/test_sessions/1e4b0840-3c0f-4c74-8d51-0005c0b498c8" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/1e4b0840-3c0f-4c74-8d51-0005c0b498c8</a></p> Related Development Rank 1|hzz6iv: Rank (Obsolete) 9223372036854775807 Severity