https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11936 Lustre <p>We upgraded our cluster Sherlock to Lustre 2.12 yesterday and put Fir (Lustre 2.12 servers) into production yesterday but this morning, the filesystem is unusable due to a super high load of ldlm threads on the MDS servers.</p> <p>I can see plenty of these on the MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Wed Feb 6 09:19:25 2019][1695265.110496] LNet: Service thread pid 35530 was inactive for 350.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Wed Feb 6 09:19:25 2019][1695265.127609] Pid: 35530, comm: mdt02_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Wed Feb 6 09:19:25 2019][1695265.137522] Call Trace: [Wed Feb 6 09:19:25 2019][1695265.140184] [&lt;ffffffffc0e3a0bd&gt;] ldlm_completion_ast+0x63d/0x920 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.147339] [&lt;ffffffffc0e3adcc&gt;] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.154723] [&lt;ffffffffc15164ab&gt;] mdt_object_local_lock+0x50b/0xb20 [mdt] [Wed Feb 6 09:19:25 2019][1695265.161731] [&lt;ffffffffc1516b30&gt;] mdt_object_lock_internal+0x70/0x3e0 [mdt] [Wed Feb 6 09:19:25 2019][1695265.168906] [&lt;ffffffffc1517d1a&gt;] mdt_getattr_name_lock+0x90a/0x1c30 [mdt] [Wed Feb 6 09:19:25 2019][1695265.176002] [&lt;ffffffffc151fbb5&gt;] mdt_intent_getattr+0x2b5/0x480 [mdt] [Wed Feb 6 09:19:25 2019][1695265.182750] [&lt;ffffffffc151ca18&gt;] mdt_intent_policy+0x2e8/0xd00 [mdt] [Wed Feb 6 09:19:25 2019][1695265.189403] [&lt;ffffffffc0e20ec6&gt;] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.196349] [&lt;ffffffffc0e498a7&gt;] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.203643] [&lt;ffffffffc0ed0302&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.210001] [&lt;ffffffffc0ed735a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.217133] [&lt;ffffffffc0e7b92b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.225045] [&lt;ffffffffc0e7f25c&gt;] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Wed Feb 6 09:19:25 2019][1695265.231563] [&lt;ffffffffadcc1c31&gt;] kthread+0xd1/0xe0 [Wed Feb 6 09:19:25 2019][1695265.236657] [&lt;ffffffffae374c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [Wed Feb 6 09:19:25 2019][1695265.243306] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [Wed Feb 6 09:19:25 2019][1695265.248518] LustreError: dumping log to /tmp/lustre-log.1549473603.35530 </pre> </div></div> <p>On Fir, we have two Lustre 2.12 MDS, <tt>fir-md1-s1</tt> and <tt>fir-md1-s2</tt>, each with 2 MDTs. I dumped the current tasks using sysrq to the console and I'm attaching the full console log for both MDS servers. The servers don't crash but lead to unsable/blocking clients,however from time to time we can access the filesystem. Any help would be appreciated. Thanks!</p> <p>Stephane</p> CentOS 7.6, Lustre 2.12.0, 3.10.0-957.1.3.el7_lustre.x86_64 LU-11936

High ldlm load, slow/unusable filesystem

Bug Blocker Reopened Unresolved Serguei Smirnov Stephane Thiell Wed, 6 Feb 2019 17:54:01 +0000 Wed, 27 Nov 2019 16:06:52 +0000 Lustre 2.12.0 0 9 <p>Ah! Just after submitting this ticket and attaching the different files, I just noticed that a possible issue could be that we do have (again) one client having a NID using @tcp leading to unusable servers. </p> <p>For example, from lustre-log.1549473603.35530:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@@ Request sent has failed due to network error: [sent 1549473603/real 1549473603] req@ffff912521032d00 x1623682053684704/t0(0) o106-&gt;fir-MDT0002@10.10.114.10@tcp:15/16 lens 296/280 e 0 to 1 dl 1549473614 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 ��lib-move.c�lnet_handle_find_routed_path�no route to 10.10.114.10@tcp from &lt;?&gt; </pre> </div></div> <p>All our Lustre servers are exclusively IB. If this is the cause of the issue, this could be a duplicate of my previous ticket <a href="https://jira.whamcloud.com/browse/LU-11888" title="Unreachable client NID confusing Lustre 2.12" class="issue-link" data-issue-key="LU-11888">LU-11888</a>.<br/> I'll investigate and get rid of this bad node and then confirm or not.<br/> Best,<br/> Stephane</p> <p>Seems related to a ticket you already have</p> <p>Yes I confirm that when we get rid of all bogus clients and restart all Lustre servers, everything is back to normal. The servers don't recover by themselves so a full restart is required apparently. Other Lustre servers running 2.8 or 2.10 are not affected. Please feel free to mark this one as duplicate of <a href="https://jira.whamcloud.com/browse/LU-11888" title="Unreachable client NID confusing Lustre 2.12" class="issue-link" data-issue-key="LU-11888">LU-11888</a>. I also opened <a href="https://jira.whamcloud.com/browse/LU-11937" title="lnet.service randomly load tcp NIDs" class="issue-link" data-issue-key="LU-11937"><del>LU-11937</del></a> to find out why in some case a NID with tcp0 is loaded on the clients.<br/> Thanks!<br/> Stephane</p> <p>ok - thanks Stephane</p> <p>This issue should probably be reopen because we just seen it with 2.12.3 RC1. After a client announced itself with a @tcp NID, it was very hard to remove it from Lustre. We had to evict the bogus tcp nid + use <tt>lnetctl peer del --prim_nid</tt>.</p> <p>And it didn't even work the first time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-md1-s1 fir-MDT0000]# lnetctl peer show --nid 10.10.117.42@tcp peer: - primary nid: 10.10.117.42@tcp Multi-Rail: True peer ni: - nid: 10.9.117.42@o2ib4 state: NA - nid: 10.10.117.42@tcp state: NA [root@fir-md1-s1 fir-MDT0000]# lnetctl peer del --prim_nid 10.10.117.42@tcp [root@fir-md1-s1 fir-MDT0000]# lnetctl peer show --nid 10.10.117.42@tcp peer: - primary nid: 10.10.117.42@tcp Multi-Rail: False peer ni: - nid: 10.10.117.42@tcp state: NA </pre> </div></div> <p>See, it removed the good one (o2ib4)!</p> <p> </p> <p>With a combination of eviction + peer deletion, it worked:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-md1-s1 fir-MDT0000]# cat exports/10.10.117.42@tcp/uuid 2f0c7362-5db5-a6c6-08eb-fa16109cc0f9 [root@fir-md1-s1 fir-MDT0000]# echo 2f0c7362-5db5-a6c6-08eb-fa16109cc0f9 &gt; evict_client [root@fir-md1-s1 fir-MDT0000]# lnetctl peer del --prim_nid 10.10.117.42@tcp [root@fir-md1-s1 fir-MDT0000]# lnetctl peer show --nid 10.9.117.42@o2ib4 peer: - primary nid: 10.9.117.42@o2ib4 Multi-Rail: True peer ni: - nid: 10.9.117.42@o2ib4 state: NA [root@fir-md1-s1 fir-MDT0000]# ls exports| grep tcp [root@fir-md1-s1 fir-MDT0000]# </pre> </div></div> <p>This is a problem, we have IB only routers and no tcp network/route defined on the servers, I don't know why Lustre does accept it at all. Also, the ldlm_bl thread at 100% is also still a problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 40268 root 20 0 0 0 0 S 100.0 0.0 114:45.76 ldlm_bl_06 16505 root 20 0 164284 4424 1548 R 22.2 0.0 0:00.04 top 35331 root 20 0 0 0 0 S 5.6 0.0 100:37.82 kiblnd_sd_03_03 35333 root 20 0 0 0 0 S 5.6 0.0 0:48.41 lnet_discovery </pre> </div></div> <p>I'm feeling hesitant of dumping the tasks right now, as the filesystem is heavily used.</p> <p>This is the task that was in an endless loop:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Tue Oct 22 06:46:06 2019][502582.891745] ldlm_bl_06 R running task 0 40268 2 0x00000080 [Tue Oct 22 06:46:06 2019][502582.898941] Call Trace: [Tue Oct 22 06:46:07 2019][502582.901568] [&lt;ffffffffc10018d9&gt;] ? ptlrpc_expire_one_request+0xf9/0x520 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.909199] [&lt;ffffffffc1003da8&gt;] ? ptlrpc_check_set.part.23+0x378/0x1df0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.916913] [&lt;ffffffffc100587b&gt;] ? ptlrpc_check_set+0x5b/0xe0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.923673] [&lt;ffffffffc1005dea&gt;] ? ptlrpc_set_wait+0x4ea/0x790 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.930470] [&lt;ffffffffaf6d7c40&gt;] ? wake_up_state+0x20/0x20 [Tue Oct 22 06:46:07 2019][502582.936184] [&lt;ffffffffc0fc3055&gt;] ? ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.943118] [&lt;ffffffffc0fc452f&gt;] ? __ldlm_reprocess_all+0x11f/0x360 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.950395] [&lt;ffffffffc0fc4783&gt;] ? ldlm_reprocess_all+0x13/0x20 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.957331] [&lt;ffffffffc0fdc01e&gt;] ? ldlm_cli_cancel_local+0x29e/0x3f0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.964708] [&lt;ffffffffc0fe1b67&gt;] ? ldlm_cli_cancel+0x157/0x620 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.971541] [&lt;ffffffffaf81bd89&gt;] ? ___slab_alloc+0x209/0x4f0 [Tue Oct 22 06:46:07 2019][502582.977440] [&lt;ffffffffc0fe20d4&gt;] ? ldlm_blocking_ast_nocheck+0xa4/0x310 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.985094] [&lt;ffffffffc0fe247a&gt;] ? ldlm_blocking_ast+0x13a/0x170 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.992129] [&lt;ffffffffc0fbc02c&gt;] ? lock_res_and_lock+0x2c/0x50 [ptlrpc] [Tue Oct 22 06:46:07 2019][502582.999006] [&lt;ffffffffc0fedcd8&gt;] ? ldlm_handle_bl_callback+0xf8/0x4f0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.006463] [&lt;ffffffffc0fc00e0&gt;] ? ldlm_lock_decref_internal+0x1a0/0xa30 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.014202] [&lt;ffffffffc0d693b9&gt;] ? class_handle2object+0xb9/0x1c0 [obdclass] [Tue Oct 22 06:46:07 2019][502583.021483] [&lt;ffffffffc0fc0a70&gt;] ? ldlm_lock_decref_and_cancel+0x80/0x150 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.029249] [&lt;ffffffffc1447835&gt;] ? mgs_completion_ast_generic+0x125/0x200 [mgs] [Tue Oct 22 06:46:07 2019][502583.036765] [&lt;ffffffffc1447930&gt;] ? mgs_completion_ast_barrier+0x20/0x20 [mgs] [Tue Oct 22 06:46:07 2019][502583.044126] [&lt;ffffffffc1447943&gt;] ? mgs_completion_ast_ir+0x13/0x20 [mgs] [Tue Oct 22 06:46:07 2019][502583.051106] [&lt;ffffffffc0fbdc58&gt;] ? ldlm_work_cp_ast_lock+0xa8/0x1d0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.058392] [&lt;ffffffffc1005972&gt;] ? ptlrpc_set_wait+0x72/0x790 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.065122] [&lt;ffffffffaf81e41d&gt;] ? kmem_cache_alloc_node_trace+0x11d/0x210 [Tue Oct 22 06:46:07 2019][502583.072240] [&lt;ffffffffc0d68a69&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [Tue Oct 22 06:46:07 2019][502583.079528] [&lt;ffffffffc0fbdbb0&gt;] ? ldlm_work_gl_ast_lock+0x3a0/0x3a0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.086915] [&lt;ffffffffc0ffc202&gt;] ? ptlrpc_prep_set+0xd2/0x280 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.093672] [&lt;ffffffffc0fc3055&gt;] ? ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.100649] [&lt;ffffffffc0fc452f&gt;] ? __ldlm_reprocess_all+0x11f/0x360 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.107927] [&lt;ffffffffc0fc52a5&gt;] ? ldlm_cancel_lock_for_export.isra.27+0x195/0x360 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.116509] [&lt;ffffffffc0fc54ac&gt;] ? ldlm_cancel_locks_for_export_cb+0x3c/0x50 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.124535] [&lt;ffffffffc0a8efb0&gt;] ? cfs_hash_for_each_relax+0x250/0x450 [libcfs] [Tue Oct 22 06:46:07 2019][502583.132112] [&lt;ffffffffc0fc5470&gt;] ? ldlm_cancel_lock_for_export.isra.27+0x360/0x360 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.140690] [&lt;ffffffffc0fc5470&gt;] ? ldlm_cancel_lock_for_export.isra.27+0x360/0x360 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.149260] [&lt;ffffffffc0a92510&gt;] ? cfs_hash_for_each_empty+0x80/0x1d0 [libcfs] [Tue Oct 22 06:46:07 2019][502583.156714] [&lt;ffffffffc0fc57ba&gt;] ? ldlm_export_cancel_locks+0xaa/0x180 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.164257] [&lt;ffffffffc0fee888&gt;] ? ldlm_bl_thread_main+0x7b8/0xa40 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.171408] [&lt;ffffffffaf6d7c40&gt;] ? wake_up_state+0x20/0x20 [Tue Oct 22 06:46:07 2019][502583.177127] [&lt;ffffffffc0fee0d0&gt;] ? ldlm_handle_bl_callback+0x4f0/0x4f0 [ptlrpc] [Tue Oct 22 06:46:07 2019][502583.184618] [&lt;ffffffffaf6c2e81&gt;] ? kthread+0xd1/0xe0 [Tue Oct 22 06:46:07 2019][502583.189804] [&lt;ffffffffaf6c2db0&gt;] ? insert_kthread_work+0x40/0x40 [Tue Oct 22 06:46:07 2019][502583.195988] [&lt;ffffffffafd77c24&gt;] ? ret_from_fork_nospec_begin+0xe/0x21 [Tue Oct 22 06:46:07 2019][502583.202696] [&lt;ffffffffaf6c2db0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>I'm attaching the full task dump when this was happening as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/33701/33701_fir-md1-s1_20191022.log" title="fir-md1-s1_20191022.log attached to LU-11936">fir-md1-s1_20191022.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>We have restarted the MGS. Unmounting it made the server unresponsive.</p> <p>Hi Stephane,</p> <p>If the @tcp NID seen on the server is a result of misconfiguration on any of the clients, then the reported behaviour with removing the primary nid without evicting beforehand may be expected as the server will keep trying to rebuild the representation of the client.</p> <p>Would it be possible to get configuration details from the server/router/client (via lnetctl export)?</p> <p>It would make it easier to understand what is causing the issue and how ldml is getting affected.</p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Serguei,</p> <p>We're seeing a similar situation and i can reproduce it and provide you details.  I added a comment to  <a href="https://jira.whamcloud.com/browse/LU-11989" title="Global filesystem hangs in 2.12" class="issue-link" data-issue-key="LU-11989"><del>LU-11989</del></a> here which includes a relevant snippet from our MDS logs: <a href="https://jira.whamcloud.com/browse/LU-11989?focusedCommentId=258916&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-258916" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11989?focusedCommentId=258916&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-258916</a></p> <p>In our example, we have four cluster nodes that initially connected to one of our 2.12 file systems using the wrong NID - one behind a NAT. The NIDs trapped behind the NAT are 10.23.x.x@tcp. </p> <p>Let me know if i can provide you any other information.</p> <p>Jesse Stroik</p> Duplicate LU-11888 Development Rank 1|i00b4n: Rank (Obsolete) 9223372036854775807 Severity