https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15517 Lustre <p>MDT fails to enter recovery when there is one MDT running on its partner MDS. The recovery_status file indicates "WAITING" and reports the MDT running on its parter as a non-ready MDT.</p> <p>The console log of the MDT unable to enter recovery repeatedly shows messages like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Thu Feb 3 12:15:18 2022] Lustre: 4102:0:(ldlm_lib.c:1827:extend_recovery_timer()) lquake-MDT0009: extended recovery timer reached hard limit: 900, extend: 1 [Thu Feb 3 12:15:18 2022] Lustre: 4102:0:(ldlm_lib.c:1827:extend_recovery_timer()) Skipped 29 previous similar messages </pre> </div></div> <p>My steps to reproduce:<br/> 1. Start all MDTs on their primary MDS (in my case, that's jet1...jet16 =&gt; MDT0000...MDT000f). Allow them to complete recovery.<br/> 2. umount MDT000e on jet15, and mount it on jet16. Allow it to complete recovery.<br/> 3. umount MDT0009 on jet10, and then mount it again (on the same node, jet10, where it was happily running moments ago).</p> <p>Debug log shows that the update log for MDT000e was not received. There are no messages in the console log regarding MDT000e after MDT0009 starts up.</p> <p>I determined the PID of the thread lod_sub_recovery_thread() for MDT000e. The debug log shows the thread starts up, follows roughly this sequence of calls, and never returns from ptlrpc_set_wait().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lod_sub_prep_llog&gt;llog_osd_get_cat_list-&gt;dt_locate_at-&gt;lu_object_find_at-&gt; ?-&gt; osp_attr_get-&gt;osp_remote_sync-&gt;ptlrpc_queue_wait-&gt;ptlrpc_set_wait </pre> </div></div> <p>So apparently the RPC never times out, so upper layers never get to retry, there are no error messages reporting the problem, and the MDT never enters recovery.</p> <p>Our patch stack is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>82ea54e (tag: 2.12.8_6.llnl) LU-13356 client: don't use OBD_CONNECT_MNE_SWAB d06f5b2 LU-15357 mdd: fix changelog context leak 543b60b LU-9964 llite: prevent mulitple group locks d776b67 LU-15234 lnet: Race on discovery queue 77040da Revert "LU-15234 lnet: Race on discovery queue" bba827c LU-15234 lnet: Race on discovery queue 7faa872 LU-14865 utils: llog_reader.c printf type mismatch 5dc104e LU-13946 build: OpenZFS 2.0 compatibility 0fef268 TOSS-4917 grant: chatty warning in tgt_grant_incoming 5b70822 TOSS-4917 grant: improve debug for grant calcs 8af02e8 log lfs setstripe paths to syslog 391be81 Don't install lustre init script on systemd systems c613926 LLNL build customizations a4f71cd TOSS-4431 build: build ldiskfs only for x86_64 067cb55 (tag: v2_12_8, tag: 2.12.8) New release 2.12.8</pre> </div></div> <p>See <a href="https://github.com/LLNL/lustre/tree/2.12.8-llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/tree/2.12.8-llnl</a> for the details.</p> 3.10.0-1160.53.1.1chaos.ch6.x86_64 <br/> zfs-0.7.11-9.8llnl.ch6.x86_64 <br/> lustre-2.12.8_6.llnl-1.ch6.x86_64 LU-15517

MDT stuck in recovery if one other MDT is failed over to partner node

Bug Critical Open Unresolved Mikhail Pershin Olaf Faaland llnl Thu, 3 Feb 2022 20:29:40 +0000 Fri, 22 Jul 2022 22:43:18 +0000 Lustre 2.12.8 0 9 <p>This appears to be new to us, which makes me a bit suspicious of the top commit, see <a href="https://jira.whamcloud.com/browse/LU-15453" title="MDT shutdown hangs on mutex_lock, possibly cld_lock" class="issue-link" data-issue-key="LU-15453">LU-15453</a> / <a href="https://jira.whamcloud.com/browse/LU-13356" title="lctl conf_param hung on the MGS node" class="issue-link" data-issue-key="LU-13356"><del>LU-13356</del></a>.  But I haven't yet proven it by reverting that patch and re-testing.</p> <p>Debug log and console log attached.</p> <p>Note there is no network or lnet issue within the cluster.  As I mentioned, all the targets mount fine on their primary MDS, and (one at a time), on their secondary MDS.  lnetctl ping works correctly in both directions.</p> <p>I've also attached the debug log and dmesg output from jet16, where both MDT000e and MDT000f are running.</p> <p>For my reference, my local ticket is TOSS5536</p> <p>Once it's entered ptlrpc_set_wait(), the debug log reports this over and over again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:13.0:1643913970.009932:0:4100:0:(client.c:2390:ptlrpc_set_wait()) set ffffa1897c264e00 going to sleep for 0 seconds 00000100:00000001:13.0:1643913970.009933:0:4100:0:(client.c:1726:ptlrpc_check_set()) Process entered 00000100:00000001:13.0:1643913970.009933:0:4100:0:(client.c:2647:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:13.0:1643913970.009934:0:4100:0:(client.c:1195:ptlrpc_import_delay_req()) Process entered 00000100:00000001:13.0:1643913970.009934:0:4100:0:(client.c:1250:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:13.0:1643913970.009935:0:4100:0:(client.c:2140:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:13.0:1643913970.009936:0:4100:0:(client.c:1726:ptlrpc_check_set()) Process entered 00000100:00000001:13.0:1643913970.009936:0:4100:0:(client.c:2647:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:13.0:1643913970.009936:0:4100:0:(client.c:1195:ptlrpc_import_delay_req()) Process entered 00000100:00000001:13.0:1643913970.009937:0:4100:0:(client.c:1250:ptlrpc_import_delay_req()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:13.0:1643913970.009937:0:4100:0:(client.c:2140:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>Mike</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>I re-tested with lustre-2.12.8_5.llnl (without the OBD_CONNECT_MNE_SWAB patch) and see the same issue.   I'll have to do a lot more testing to determine how far back this behavior was introduced, if that's necessary.</p> <p>I tested with lustre-2.12.7_2.llnl-2.ch6.x86_64 and reproduced the issue, so it's not new.  See <a href="https://github.com/LLNL/lustre/releases/tag/2.12.7_2.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases/tag/2.12.7_2.llnl</a> for the code.  Patch stack is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>* f372cfb (tag: 2.12.7_2.llnl) LU-14865 utils: llog_reader.c printf type mismatch * 5b7709c LU-14733 o2iblnd: Avoid double posting invalidate * cdde64a LU-14733 o2iblnd: Move racy NULL assignment * 38fade1 LU-12836 osd-zfs: Catch all ZFS pool change events * c7d6ecf LU-13946 build: OpenZFS 2.0 compatibility * ec7dd98 TOSS-4917 grant: chatty warning in tgt_grant_incoming * eb6d90f TOSS-4917 grant: improve debug for grant calcs * 4f202a3 log lfs setstripe paths to syslog * 939ba8a Don't install lustre init script on systemd systems * 3cfca06 LLNL build customizations * ce7af9a TOSS-4431 build: build ldiskfs only for x86_64 * 6030d0c (tag: v2_12_7, tag: 2.12.7) New release 2.12.7</pre> </div></div> <p>The amount of time I waited in this test was 20 minutes. I don't recall how long I waited in my earlier experiments.</p> <p>I reproduced with lustre-2.12.5_10.llnl</p> <p>Ping</p> <p>Olaf, are you able to reproduce that on clean system or it is reproduced on the working cluster? In the last case, could you collect update llog info from the MDT with stuck recovery? (jet16 in your last example). You can use <tt>debugfs</tt> for that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # debugfs -c &lt;device&gt; -R <span class="code-quote">"ls -l update_log_dir/"</span> &gt; jet16_update_log_dir.txt</pre> </div></div> <p> </p> <p>Thanks</p> <p>Hi Mikhail,<br/> Sorry for the delay. Good question - I was not able to reproduce on a clean system (I thought I'd tried that, but apparently not).</p> <p>I reproduced on jet again. In this instance, lquake-MDT000e is failed over and running on jet16, when it normally would be on jet15. lquake-MDT0009 was umounted and remounted on jet10, and is stuck in WAITING.</p> <p>NIDs are as follows, in case it helps interpret what you see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ejet10: 172.19.1.120@o2ib100 ejet15: 172.19.1.125@o2ib100 ejet16: 172.19.1.126@o2ib100 </pre> </div></div> <p>The file containing the contents of update_log_dir/ is 2.1GB compressed. Do you still want me to send it to you? If not, do you want me to run llog_reader on each file and collect the results instead? The ones I spot-checked reported 0 records, even when they were large. Here's an example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@jet10:update_log_dir]# stat [0x40002d2c5:0x2:0x0] File: '[0x40002d2c5:0x2:0x0]' Size: 938979232 Blocks: 14913761 IO Block: 512 regular file Device: 31h/49d Inode: 28181515 Links: 1 Access: (0644/-rw-r--r--) Uid: (56288/ UNKNOWN) Gid: (56288/ UNKNOWN) Access: 2018-02-23 08:14:08.000000000 -0800 Modify: 2018-02-23 08:14:08.000000000 -0800 Change: 2018-02-23 08:14:08.000000000 -0800 Birth: - [root@jet10:update_log_dir]# llog_reader [0x40002d2c5:0x2:0x0] Header size : 32768 llh_size : 472 Time : Fri Feb 23 08:14:08 2018 Number of records: 0 cat_idx: 13 last_idx: 161679 Target uuid : ----------------------- </pre> </div></div> <p>thanks,<br/> Olaf</p> <p>I ran llog_reader on each of the files in update_log_dir and attached the output in the file named lu-15517.llog_reader.out.</p> <p>The files with Number of records &gt; 0 are:<br/> <span class="error">&#91;0x400000401:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000402:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000403:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000404:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000405:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000406:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000407:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000408:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x400000409:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040a:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040b:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040c:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040d:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040e:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x40000040f:0x1:0x0&#93;</span></p> <p>Do you want me to do something with the files in the OI they refer to?</p> <p>Olaf, please check these files with llog_reader. I wonder if any of them contains 'holes' in records but llog_reader in your version of Lustre might not be able to translate update records correctly and report them as 'Unknown' type, you can check.  I'd ask you to upload these files listed above for analysis here, is that possible?</p> <p>ah, missed that you've uploaded llog_reader output. Looks like it knows about update log, let me check it first</p> <p>Olaf, no need to upload all llogs right now, let me think a bit first</p> <p>Olaf, the reason why I've asked about reproducing that on clean system is that there were several issues with update log processing causing MDT-MDT recovery to fail. Could you check if any of MDT reported errors from <tt>lod_sub_recovery_thread</tt> or <tt>llog_process_thread()</tt> or any other llog processing complains when MDT is stuck in WAITING state? </p> <p>Hi Mikhail,<br/> I found that there are several errors reported in lod_sub_process_config().  They were not encountered during my experiment, but earlier in the day today when I stopped the targets prior to a software update.  Note they appeared on several nodes, all MDTs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 11 14:46:58 jet2 kernel: LustreError: 27555:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0001-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet3 kernel: LustreError: 15820:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0002-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet5 kernel: LustreError: 5267:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0004-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet7 kernel: LustreError: 22882:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0006-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet8 kernel: LustreError: 21185:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0007-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet9 kernel: LustreError: 3160:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0008-mdtlov: error cleaning up LOD index 1: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet10 kernel: LustreError: 18778:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0009-mdtlov: error cleaning up LOD index 1: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet12 kernel: LustreError: 3992:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT000b-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet14 kernel: LustreError: 1060:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT000d-mdtlov: error cleaning up LOD index 1: cmd 0xcf031 : rc = -19 Apr 11 14:46:58 jet15 kernel: LustreError: 22650:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT000e-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 </pre> </div></div> <p>I also saw these errors on jet6 (which was hosting MDT0005):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-04-11 14:48:39 [1649713719.122533] LustreError: 30035:0:(osp_object.c:594:osp_attr_get()) lquake-MDT0000-osp-MDT0005:osp_attr_get update error [0x20000000a:0x0:0x0]: rc = -5 2022-04-11 14:48:39 [1649713719.141533] LustreError: 30035:0:(llog_cat.c:447:llog_cat_close()) lquake-MDT0000-osp-MDT0005: failure destroying log during cleanup: rc = -5 2022-04-11 14:48:39 [1649713719.467531] Lustre: server umount lquake-MDT0005 complete </pre> </div></div> <p>In all cases, the lod_sub_process_config() error was preceeded by a osp_disconnect() error, like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-04-11 14:46:58 [1649713618.877752] Lustre: Failing over lquake-MDT0004 2022-04-11 14:46:58 [1649713618.896752] LustreError: 11-0: lquake-MDT0001-osp-MDT0004: operation mds_disconnect to node 172.19.1.112@o2ib100 failed: rc = -107 2022-04-11 14:46:58 [1649713618.915752] Lustre: lquake-MDT000e-osp-MDT0004: Connection to lquake-MDT000e (at 172.19.1.125@o2ib100) was lost; in progress operations using this service will wait for recovery to complete 2022-04-11 14:46:58 [1649713618.936752] LustreError: 5267:0:(osp_dev.c:485:osp_disconnect()) lquake-MDT0003-osp-MDT0004: can't disconnect: rc = -19 2022-04-11 14:46:58 [1649713618.950752] LustreError: 5267:0:(lod_dev.c:267:lod_sub_process_config()) lquake-MDT0004-mdtlov: error cleaning up LOD index 3: cmd 0xcf031 : rc = -19 </pre> </div></div> <p>Olaf, I'd propose to add several patches to your stack. I think the problem in general is that update llogs became quite big at some or several MDTs. There are series of patches to manage update llogs more gracefully (top to bottom):</p> <ul> <li><a href="https://review.whamcloud.com/47011" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47011</a> - <a href="https://jira.whamcloud.com/browse/LU-15645" title="gap in recovery llog should not be a fatal error" class="issue-link" data-issue-key="LU-15645"><del>LU-15645</del></a> obdclass: llog to handle gaps</li> <li><a href="https://review.whamcloud.com/47010" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47010</a> - <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: osp_send_update_req() should check generation</li> <li><a href="https://review.whamcloud.com/45847" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45847</a> - <a href="https://jira.whamcloud.com/browse/LU-12577" title="chlg_load failed to process llog -2 or -5 on client" class="issue-link" data-issue-key="LU-12577"><del>LU-12577</del></a> llog: protect partial updates from readers</li> <li><a href="https://review.whamcloud.com/46863" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46863</a> - <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> osp: invalidate object on write error</li> <li><a href="https://review.whamcloud.com/46873" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46873</a>  - mostly to make upper one to apply cleanly</li> </ul> <p>This might help MDT to process update logs after all if problem is in them as I am expecting. Another patch that could also be useful is about OSP_DISCONNECT handling between MDTs:</p> <p><a href="https://review.whamcloud.com/44753" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44753</a> - osp: do force disconnect if import is not ready</p> <p>Mike</p> <p> </p> Development Rank 1|i02h87: Rank (Obsolete) 9223372036854775807 Severity