https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12102 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/8b3b3134-48aa-11e9-b98a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8b3b3134-48aa-11e9-b98a-52540065bddc</a></p> <p>test_7 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Update not seen after 6s: wanted 'scanning' got 'completed' (8) Expected 'scanning' on mds1 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-scrub test_7 - (8) Expected 'scanning' on mds1</p> LU-12102

sanity-scrub test_7: (8) Expected 'scanning' on mds1

Bug Minor Open Unresolved WC Triage Maloo Mon, 25 Mar 2019 07:01:32 +0000 Tue, 21 Feb 2023 16:43:45 +0000 Lustre 2.13.0 0 2 <p>Some notes.  It looks like we have trouble reconnecting the MDTs, and that causes this message (from dmesg on MDS1):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[16447.097886] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200001b74:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. </pre> </div></div> <p>This makes the OI scrub startup weird, and the subsequent messages in the log are different from the success case.</p> <p>We're having trouble re-establishing the connection here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:02020000:0.0:1552798330.508523:0:17624:0:(client.c:1279:ptlrpc_check_status()) 11-0: lustre-MDT0000-osp-MDT0002: operation mds_connect to node 0@lo failed: rc = -114 00000100:00080000:0.0:1552798330.511162:0:17624:0:(import.c:1338:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 10.9.6.9@tcp failed (-114) 00000100:02020000:0.0:1552798330.512119:0:17624:0:(client.c:1279:ptlrpc_check_status()) 11-0: lustre-MDT0001-osp-MDT0002: operation mds_connect to node 10.9.6.10@tcp failed: rc = -114 00000100:00080000:0.0:1552798330.512123:0:17624:0:(import.c:1338:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 10.9.6.10@tcp failed (-114) </pre> </div></div> <p>This is before we turn on oi scub:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000001:02000400:0.0:1552798342.891701:0:29330:0:(debug.c:511:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl set_param -n osd-*.*.auto_scrub=1 </pre> </div></div> <p>But we're still taking errors after that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1552798343.266169:0:26241:0:(service.c:2198:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt_out00_000:lustre-MDT0001-mdtlov_UUID+5:14884:x1628209949343232:12345-10.9.6.10@tcp:1000 Request processed in 3455us (3476us total) trans 0 rc -115/-115 </pre> </div></div> <p>That's an OUT_UPDATE request.</p> <p>(Posting this to get it out, more to check...)</p> <p>I dug in some more &amp; didn't find any smoking guns - I'm going to drop this for now.  Perhaps helpful for someone else or just something for later.</p> <p>But the cause of the failure definitely seems to be the MDSes not being fully connected, leading to this error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[16447.097886] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200001b74:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. </pre> </div></div> <p>This suggests we may need some better check for startup, as this does appear to unwind itself, it just does so after we start oi scrub.</p> Development Rank 1|i00dsv: Rank (Obsolete) 9223372036854775807 Severity