https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16408 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/bd566ffb-4675-4852-9ac7-ec33bd93b99f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/bd566ffb-4675-4852-9ac7-ec33bd93b99f</a></p> <p>test_33 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Started lustre-MDT0000 Starting client: trevis-83vm7.trevis.whamcloud.com: -o user_xattr,flock trevis-80vm7@tcp:/lustre /mnt/lustre2 CMD: trevis-83vm7.trevis.whamcloud.com mkdir -p /mnt/lustre2 CMD: trevis-83vm7.trevis.whamcloud.com mount -t lustre -o user_xattr,flock trevis-80vm7@tcp:/lustre /mnt/lustre2 </pre> </div></div> <p>Dmesg on client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 11-0: lustre-MDT0000-mdc-ffff9aa6d17cc800: operation mds_connect to node 10.240.42.182@tcp failed: rc = -11 LustreError: Skipped 3 previous similar messages Lustre: lustre-OST0006-osc-ffff9aa6d92e2800: Connection to lustre-OST0006 (at 10.240.42.196@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0001-osc-ffff9aa6d92e2800: Connection restored to (at 10.240.42.196@tcp) Lustre: Skipped 5 previous similar messages LustreError: 10874:0:(lmv_obd.c:1287:lmv_statfs()) lustre-MDT0000-mdc-ffff9aa6d17cc800: can't stat MDS #0: rc = -11 Lustre: Unmounted lustre-client LustreError: 10874:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount &lt;unknown&gt;: rc = -11 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-dual test_33 - Timeout occurred after 434 minutes, last suite running was replay-dual</p> LU-16408

replay-dual test_33: unable to mount /mnt/lustre2

Bug Minor Resolved Fixed Etienne Aujames Maloo Thu, 15 Dec 2022 20:31:50 +0000 Fri, 27 Oct 2023 02:39:53 +0000 Sat, 23 Sep 2023 14:10:16 +0000 Lustre 2.16.0 Lustre 2.16.0 0 7 <p>The replay-dual test_33 was added by <a href="https://review.whamcloud.com/48082" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48082</a> ("<a href="https://jira.whamcloud.com/browse/LU-15935" title="MDT mount fails with &quot;duplicate generation for client export&quot; during failover" class="issue-link" data-issue-key="LU-15935"><del>LU-15935</del></a> target: keep track of multirpc slots in last_rcvd").<br/> Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=eaujames" class="user-hover" rel="eaujames">eaujames</a>, could you please advise?</p> <p>Hi,</p> <p>The mounted client did not finish completely its recovery and was evicted in REPLAY_LOCKS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Tue Nov 15 16:29:06 2022] LustreError: 876057:0:(mdt_handler.c:7441:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [Tue Nov 15 16:29:06 2022] LustreError: 876057:0:(ldlm_lib.c:2888:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [Tue Nov 15 16:29:06 2022] Lustre: 874819:0:(ldlm_lib.c:2294:target_recovery_overseer()) recovery is aborted, evict exports in recovery [Tue Nov 15 16:29:06 2022] Lustre: 874819:0:(ldlm_lib.c:2294:target_recovery_overseer()) Skipped 2 previous similar messages [Tue Nov 15 16:29:06 2022] Lustre: lustre-MDT0000: disconnecting 2 stale clients [Tue Nov 15 16:29:06 2022] LustreError: 874819:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 4194304 != fo_tot_granted 6291456 [Tue Nov 15 16:29:06 2022] LustreError: 874819:0:(ldlm_lib.c:1829:abort_lock_replay_queue()) @@@ aborted: req@00000000b51e723d x1749569008570176/t0(0) o101-&gt;1b7e53c5-3301-41c4-8e4d-aab3eade9ae8@10.240.42.242@tcp:151/0 lens 328/0 e 0 to 0 dl 1668529766 ref 1 fl Complete:/40/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [Tue Nov 15 16:29:06 2022] LustreError: 874819:0:(ldlm_lib.c:1829:abort_lock_replay_queue()) Skipped 25 previous similar messages [Tue Nov 15 16:29:06 2022] Lustre: lustre-MDT0000: Denying connection for new client 1b7e53c5-3301-41c4-8e4d-aab3eade9ae8 (at 10.240.42.242@tcp), waiting for 4 known clients (2 recovered, 0 in progress, and 2 evicted) to recover in 1:14 [Tue Nov 15 16:29:06 2022] Lustre: Skipped 3 previous similar messages </pre> </div></div> <p>But the client import is set to FULL, the MDT recovery did not finish and then the MDT was unmounted. After remounting the MDT the 2sd client is unable to remount.</p> <p>This looks like a new bug. For now, I will stabilize the test.</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50434" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50434</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16408" title="replay-dual test_33: unable to mount /mnt/lustre2" class="issue-link" data-issue-key="LU-16408"><del>LU-16408</del></a> tests: fix replay-dual test 33<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b513f684834cf0fc7ebab6b319fc2ae098ff60d6</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50434/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50434/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16408" title="replay-dual test_33: unable to mount /mnt/lustre2" class="issue-link" data-issue-key="LU-16408"><del>LU-16408</del></a> tests: fix replay-dual test 33<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7f89e8c8975fcc82983c5756438861d66e64ec23</p> <p>Landed for 2.16</p> <p>the patch breaks replay-dual/33 on a local (single VM) setup. before it took ~30 seconds to complete replay-dual, now it can't complete within 30 minutes.</p> <p>replay-dual/33 gets stuck at the following operation:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ! combined_mgs_mds || $LCTL get_param mdc.*.ping || <span class="code-keyword">true</span> </pre> </div></div> <p>can you please explain the purpose of this get_param ?</p> <p>actually it mount_facet mds1 fails:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 122.710153] LustreError: 15c-8: MGC192.168.127.51@tcp: Confguration from log lustre-MDT0000 failed from MGS -5. Communication error between node &amp; MGS, a bad configuration, or other errors. See syslog <span class="code-keyword">for</span> more info [ 122.715086] LustreError: 7365:0:(tgt_mount.c:1524:server_start_targets()) failed to start server lustre-MDT0000: -5 [ 122.715704] LustreError: 7365:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -5 [ 122.716325] LustreError: 7365:0:(tgt_mount.c:1752:server_put_super()) no obd lustre-MDT0000 [ 122.718737] Lustre: server umount lustre-MDT0000 complete [ 122.719112] LustreError: 7365:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount &lt;unknown&gt;: rc = -5 mount.lustre: mount /dev/mapper/mds1_flakey at /mnt/lustre-mds1 failed: Input/output error Is the MGS running? Start of /dev/mapper/mds1_flakey on mds1 failed 5 </pre> </div></div> <p>Yes, for combined mgt/mdt, when we umount the target (failover) we disable all mgs services. So the IR (Imperative Recovery) is disabled, then the targets states are not directly updated. In the real world it has to wait for the next pinger ping (obd_timeout/4 s).<br/> The "ping" here is workarround to force update the import state.</p> <p>I think the issue here is that if the client is the on the MGS, it uses a pingless connection on 0@lo.<br/> We have to skip this test for single node testing.</p> <p>Do you have a failed test link with logs?</p> Duplicate Related LU-17229 Development Rank 1|i0384n: Rank (Obsolete) 9223372036854775807 Severity