https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15453 Lustre <p>LNet issues (See <a href="https://jira.whamcloud.com/browse/LU-15234" title="LNet high peer reference counts inconsistent with queue" class="issue-link" data-issue-key="LU-15234"><del>LU-15234</del></a> and <a href="https://jira.whamcloud.com/browse/LU-14026" title="symptoms of message loss or corruption after upgrading routers to lustre 2.12.5" class="issue-link" data-issue-key="LU-14026">LU-14026</a>) result in clients and lustre servers reporting via console logs that they lost connection to the MGS.</p> <p>We are working on solving the LNet issues, but this may also be revealing error-path issues that should be fixed.</p> <p>MDT0, which is usually running on the same server as the MGS, is one of the targets which reports a lost connection (they are separate devices, stored in distinct datasets, started/stopped separately):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MGC172.19.3.98@o2ib600: Connection to MGS (at 0@lo) was lost </pre> </div></div> <p>Attempting to shutdown the MDT hangs, with this stack reported by the watchdog:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> schedule_preempt_disabled+0x39/0x90 __mutex_lock_slowpath+0x10f/0x250 mutex_lock+0x32/0x42 mgc_process_config+0x21a/0x1420 [mgc] obd_process_config.constprop.14+0x75/0x210 [obdclass] ? lprocfs_counter_add+0xf9/0x160 [obdclass] lustre_end_log+0x1ff/0x550 [obdclass] server_put_super+0x82e/0xd00 [obdclass] generic_shutdown_super+0x6d/0x110 kill_anon_super+0x12/0x20 lustre_kill_super+0x32/0x50 [obdclass] deactivate_locked_super+0x4e/0x70 deactivate_super+0x46/0x60 cleanup_mnt+0x3f/0x80 __cleanup_mnt+0x12/0x20 task_work_run+0xbb/0xf0 do_notify_resume+0xa5/0xc0 int_signal+0x12/0x17 </pre> </div></div> <p>The server was crashed and a dump collected.  The stacks for the umount process and the ll_cfg_requeue process both have pointers to the "ls1-mdtir" config_llog_data structure; I believe cld-&gt;cld_lock is held by ll_cfg_requeue and umount is waiting on it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 4504 TASK: ffff8e8c9edc8000 CPU: 24 COMMAND: "ll_cfg_requeue" #0 [ffff8e8ac474f970] __schedule at ffffffff9d3b6788 #1 [ffff8e8ac474f9d8] schedule at ffffffff9d3b6ce9 #2 [ffff8e8ac474f9e8] schedule_timeout at ffffffff9d3b4528 #3 [ffff8e8ac474fa98] ldlm_completion_ast at ffffffffc14ac650 [ptlrpc] #4 [ffff8e8ac474fb40] ldlm_cli_enqueue_fini at ffffffffc14ae83f [ptlrpc] #5 [ffff8e8ac474fbf0] ldlm_cli_enqueue at ffffffffc14b10d1 [ptlrpc] #6 [ffff8e8ac474fca8] mgc_enqueue at ffffffffc0fb94cf [mgc] #7 [ffff8e8ac474fd70] mgc_process_log at ffffffffc0fbf393 [mgc] #8 [ffff8e8ac474fe30] mgc_requeue_thread at ffffffffc0fc1b10 [mgc] #9 [ffff8e8ac474fec8] kthread at ffffffff9cccb221 </pre> </div></div> <p>I can provide console logs and the crash dump.  I do not have lustre debug logs.</p> lustre-2.12.7_2.llnl-2.ch6.x86_64 <br/> zfs-0.7.11-9.8llnl.ch6.x86_64 <br/> 3.10.0-1160.45.1.1chaos.ch6.x86_64 LU-15453

MDT shutdown hangs on mutex_lock, possibly cld_lock

Bug Major Open Unresolved Mikhail Pershin Olaf Faaland llnl Fri, 14 Jan 2022 19:43:49 +0000 Fri, 22 Jul 2022 22:42:56 +0000 0 7 <p>For my records, my internal ticket is TOSS5512</p> <p>Serguei</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>Actually, perhaps Mike is a more appropriate candidate...</p> <p>Hi Olaf, could you please attach the stack traces of running processes at the time of the hang ("bt" from the crashdump).</p> <p>Hi, sorry for the delay.  I've attached:<br/> "bt -a" output in bt.a.txt (stack traces of the active task on each CPU)<br/> "foreach bt" output in foreach.bt.txt (stack traces of all processes)</p> <p>Symptoms remind me ticket <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a>, the related patch is not yet landed in b2_12: <a href="https://review.whamcloud.com/41309" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41309</a> </p> <p>Another thought is <a href="https://jira.whamcloud.com/browse/LU-15020" title="OSP_DISCONNECT blocking MDT unmount" class="issue-link" data-issue-key="LU-15020"><del>LU-15020</del></a> which is about the waiting for OST_DISCONNECT, but the first one looks closer to what we have here</p> <p>Hi Mikhail,</p> <p>Yes, it does look a lot like <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a>.  I see Etienne's comment about change #41309 removing interop support with v2.2 clients and servers, and that the patch therefore cannot be landed to b2_12. </p> <ol> <li>At our site, we we have only Lustre 2.10.8 routers and Lustre {2.12.8,2.14) clients/servers/routers.  We do not have v2.2 running anywhere.  Can we safely add that patch to our stack?  It would be useful to hear back about this today, if possible.</li> <li>If change #41309 cannot be landed to b2_12, what are some other options?  This question is not as urgent.</li> <li>If we see this symptom again before we have any patches landed to address it, is there other information I can gather that would help confirm this theory?</li> </ol> <p>thanks</p> <p>Olaf, the patch can be added to your stack if there is no need for 2.2 interop. As for question #2 - do you mean will there be an alternative solution in b2_12?</p> <p>As for other information to collect, it seems we can only rely on symptoms here, since related code has no any debug messages directly connected with the situation</p> <p>Mikhail,</p> <p>&gt; As for question #2 - do you mean will there be an alternative solution in b2_12?</p> <p>Yes, that was my question.</p> <p>thanks!</p> <p>Honestly it is a bit ridiculous to not land change 41309 to b2_12 at this time because of compat issue with old Lustre 2.2. Without this patch, the MGS on 2.12.x is not stable, even in a full 2.12 environment. We have patched all our clients and servers with it (we're running 2.12.x everywhere now, mostly 2.12.7 and now deploying 2.12.8 that also requires patching). Just saying. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sthiell" class="user-hover" rel="sthiell">sthiell</a>, I don't think anyone is <em>against</em> landing 41309 on b2_12 because of 2.2 interop, just that it hasn't landed yet.</p> <p>OK! Thanks Andreas!</p> <p>Stephane,</p> <p>Do you have any other patches in your stack related to recovery?</p> <p>thanks</p> <p>Hi Olaf,</p> <p>I don't think so. Our servers are running 2.12.7 with:</p> <ul> <li><a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a> client: don't use OBD_CONNECT_MNE_SWAB (41309)</li> <li><a href="https://jira.whamcloud.com/browse/LU-14688" title="Changelog cancel improvement" class="issue-link" data-issue-key="LU-14688"><del>LU-14688</del></a> mdt: changelog purge deletes plain llog (43990)</li> </ul> <p>Our clients are now slowly moving to 2.12.8 + <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a></p> Related LU-14026 LU-13356 LU-15234 Development Rank 1|i02f5j: Rank (Obsolete) 9223372036854775807 Severity