https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10800 Lustre <p>Mounts frequently hang on clients. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Mar 9 18:14:41 soak-36 kernel: INFO: task mount.lustre:2807 blocked <span class="code-keyword">for</span> more than 120 seconds. Mar 9 18:14:41 soak-36 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Mar 9 18:14:41 soak-36 kernel: mount.lustre D ffff88085b7a0000 0 2807 2806 0x00000080 Mar 9 18:14:41 soak-36 kernel: Call Trace: Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff816ab8a9&gt;] schedule+0x29/0x70 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff816a92b9&gt;] schedule_timeout+0x239/0x2c0 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff81050b4c&gt;] ? native_smp_send_reschedule+0x4c/0x70 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff810c2358&gt;] ? resched_curr+0xa8/0xc0 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff810c30d8&gt;] ? check_preempt_curr+0x78/0xa0 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff810c3119&gt;] ? ttwu_do_wakeup+0x19/0xd0 Mar 9 18:14:41 soak-36 kernel: [&lt;ffffffff816abc5d&gt;] wait_for_completion+0xfd/0x140 Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffff810c6620&gt;] ? wake_up_state+0x20/0x20 Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0b28854&gt;] llog_process_or_fork+0x244/0x450 [obdclass] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0b28a74&gt;] llog_process+0x14/0x20 [obdclass] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0b5b1c5&gt;] class_config_parse_llog+0x125/0x350 [obdclass] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc06501c8&gt;] mgc_process_cfg_log+0x788/0xc40 [mgc] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0652243&gt;] mgc_process_log+0x3d3/0x8b0 [mgc] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0b63240&gt;] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0652968&gt;] ? do_config_log_add+0x248/0x580 [mgc] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0653840&gt;] mgc_process_config+0x890/0x13f0 [mgc] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0b66c85&gt;] lustre_process_log+0x2d5/0xae0 [obdclass] Mar 9 18:14:42 soak-36 kernel: [&lt;ffffffffc0855e27&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffffc0f3e3bb&gt;] ll_fill_super+0x45b/0x1100 [lustre] Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffffc0b6caa6&gt;] lustre_fill_super+0x286/0x910 [obdclass] Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffffc0b6c820&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff81206abd&gt;] mount_nodev+0x4d/0xb0 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffffc0b64ab8&gt;] lustre_mount+0x38/0x60 [obdclass] Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff81207549&gt;] mount_fs+0x39/0x1b0 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff81224177&gt;] vfs_kern_mount+0x67/0x110 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff81226683&gt;] do_mount+0x233/0xaf0 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff811894ee&gt;] ? __get_free_pages+0xe/0x40 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff812272c6&gt;] SyS_mount+0x96/0xf0 Mar 9 18:14:43 soak-36 kernel: [&lt;ffffffff816b89fd&gt;] system_call_fastpath+0x16/0x1b Mar 9 18:16:43 soak-36 kernel: INFO: task mount.lustre:2807 blocked <span class="code-keyword">for</span> more than 120 seconds. Mar 9 18:16:43 soak-36 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. </pre> </div></div> <p>I dumped the lustre log during the hang, attached. I also crash-dumped the client, files available on soak </p> Soak stress cluster, lustre-master-next-ib build 1 LU-10800

Mount hangs on clients.

Bug Blocker Resolved Fixed Amir Shehata Cliff White soak Fri, 9 Mar 2018 18:28:25 +0000 Mon, 19 Mar 2018 17:39:20 +0000 Mon, 19 Mar 2018 17:39:20 +0000 Lustre 2.11.0 Lustre 2.11.0 0 10 <p>Lustre: Build Version: 2.10.58_140_ge103776</p> <p>Tried lustre-master-ib build 58, per Peter - same issue version=2.10.58_118_gcf6b8f5</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 842.767852] INFO: task mount.lustre:2760 blocked for more than 120 seconds. [ 842.791128] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 842.817070] mount.lustre D ffff88105acb0000 0 2760 2759 0x00000080 [ 842.840464] Call Trace: [ 842.848565] [&lt;ffffffff810cae08&gt;] ? __enqueue_entity+0x78/0x80 [ 842.867884] [&lt;ffffffff810d17fc&gt;] ? enqueue_entity+0x26c/0xb60 [ 842.887205] [&lt;ffffffff816ab8a9&gt;] schedule+0x29/0x70 [ 842.903644] [&lt;ffffffff816a92b9&gt;] schedule_timeout+0x239/0x2c0 [ 842.922960] [&lt;ffffffff810c95d5&gt;] ? sched_clock_cpu+0x85/0xc0 [ 842.941987] [&lt;ffffffff810c30d8&gt;] ? check_preempt_curr+0x78/0xa0 [ 842.961880] [&lt;ffffffff810c3119&gt;] ? ttwu_do_wakeup+0x19/0xd0 [ 842.980620] [&lt;ffffffff816abc5d&gt;] wait_for_completion+0xfd/0x140 [ 843.000516] [&lt;ffffffff810c6620&gt;] ? wake_up_state+0x20/0x20 [ 843.019150] [&lt;ffffffffc0a24854&gt;] llog_process_or_fork+0x244/0x450 [obdclass] [ 843.042950] [&lt;ffffffffc0a24a74&gt;] llog_process+0x14/0x20 [obdclass] [ 843.063850] [&lt;ffffffffc0a56da5&gt;] class_config_parse_llog+0x125/0x350 [obdclass] [ 843.088364] [&lt;ffffffffc05a51c8&gt;] mgc_process_cfg_log+0x788/0xc40 [mgc] [ 843.110283] [&lt;ffffffffc05a7243&gt;] mgc_process_log+0x3d3/0x8b0 [mgc] [ 843.131181] [&lt;ffffffffc0a5ee20&gt;] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [ 843.156774] [&lt;ffffffffc05a7968&gt;] ? do_config_log_add+0x248/0x580 [mgc] [ 843.178634] [&lt;ffffffffc05a8840&gt;] mgc_process_config+0x890/0x13f0 [mgc] [ 843.200574] [&lt;ffffffffc0a62865&gt;] lustre_process_log+0x2d5/0xae0 [obdclass] [ 843.223581] [&lt;ffffffffc0939e27&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 843.245497] [&lt;ffffffffc0ea994b&gt;] ll_fill_super+0x45b/0x1100 [lustre] [ 843.266861] [&lt;ffffffffc0a68686&gt;] lustre_fill_super+0x286/0x910 [obdclass] [ 843.289656] [&lt;ffffffffc0a68400&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [ 843.314667] [&lt;ffffffff81206abd&gt;] mount_nodev+0x4d/0xb0 [ 843.331977] [&lt;ffffffffc0a60698&gt;] lustre_mount+0x38/0x60 [obdclass] [ 843.352676] [&lt;ffffffff81207549&gt;] mount_fs+0x39/0x1b0 [ 843.369355] [&lt;ffffffff81224177&gt;] vfs_kern_mount+0x67/0x110 [ 843.387756] [&lt;ffffffff81226683&gt;] do_mount+0x233/0xaf0 [ 843.404720] [&lt;ffffffff811894ee&gt;] ? __get_free_pages+0xe/0x40 [ 843.423693] [&lt;ffffffff812272c6&gt;] SyS_mount+0x96/0xf0 [ 843.440374] [&lt;ffffffff816b89fd&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I got onto soak-34 while it was stuck in mount and found the following stack trace for the llog processing thread:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13498 llog_process_th [&lt;ffffffffc099d61b&gt;] lnet_discover_peer_locked+0x10b/0x380 [lnet] [&lt;ffffffffc099d900&gt;] LNetPrimaryNID+0x70/0x1a0 [lnet] [&lt;ffffffffc0c1937e&gt;] ptlrpc_connection_get+0x3e/0x450 [ptlrpc] [&lt;ffffffffc0c0da1c&gt;] ptlrpc_uuid_to_connection+0xec/0x1a0 [ptlrpc] [&lt;ffffffffc0bf21b2&gt;] import_set_conn+0xb2/0x7a0 [ptlrpc] [&lt;ffffffffc0bf35f9&gt;] client_obd_setup+0xd39/0x1450 [ptlrpc] [&lt;ffffffffc0d83c0b&gt;] osc_setup_common+0x5b/0x330 [osc] [&lt;ffffffffc0d83f13&gt;] osc_setup+0x33/0x2f0 [osc] [&lt;ffffffffc0d8f7c5&gt;] osc_device_alloc+0xa5/0x240 [osc] [&lt;ffffffffc0a2be2a&gt;] obd_setup+0x11a/0x2b0 [obdclass] [&lt;ffffffffc0a2d188&gt;] class_setup+0x2a8/0x840 [obdclass] [&lt;ffffffffc0a30662&gt;] class_process_config+0x1ae2/0x27b0 [obdclass] [&lt;ffffffffc0a32e94&gt;] class_config_llog_handler+0x914/0x1330 [obdclass] [&lt;ffffffffc09f69b9&gt;] llog_process_thread+0x839/0x1560 [obdclass] [&lt;ffffffffc09f80ef&gt;] llog_process_thread_daemonize+0x9f/0xe0 [obdclass] [&lt;ffffffff810b4031&gt;] kthread+0xd1/0xe0 [&lt;ffffffff816c0577&gt;] ret_from_fork+0x77/0xb0 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff 13473 mount.lustre [&lt;ffffffffc09f7924&gt;] llog_process_or_fork+0x244/0x450 [obdclass] [&lt;ffffffffc09f7b44&gt;] llog_process+0x14/0x20 [obdclass] [&lt;ffffffffc0a2a495&gt;] class_config_parse_llog+0x125/0x350 [obdclass] [&lt;ffffffffc0652038&gt;] mgc_process_cfg_log+0x788/0xc40 [mgc] [&lt;ffffffffc0655233&gt;] mgc_process_log+0x3d3/0x8b0 [mgc] [&lt;ffffffffc0656830&gt;] mgc_process_config+0x890/0x13f0 [mgc] [&lt;ffffffffc0a35fd8&gt;] lustre_process_log+0x2d8/0xaf0 [obdclass] [&lt;ffffffffc0e835eb&gt;] ll_fill_super+0x45b/0x1100 [lustre] [&lt;ffffffffc0a3be3c&gt;] lustre_fill_super+0x28c/0x920 [obdclass] [&lt;ffffffff8120948f&gt;] mount_nodev+0x4f/0xb0 [&lt;ffffffffc0a33df8&gt;] lustre_mount+0x38/0x60 [obdclass] [&lt;ffffffff81209f1e&gt;] mount_fs+0x3e/0x1b0 [&lt;ffffffff81226d57&gt;] vfs_kern_mount+0x67/0x110 [&lt;ffffffff81229263&gt;] do_mount+0x233/0xaf0 [&lt;ffffffff81229ea6&gt;] SyS_mount+0x96/0xf0 [&lt;ffffffff816c0715&gt;] system_call_fastpath+0x1c/0x21 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>This is at the <tt>schedule()</tt> call in the <tt>for (;;&#41;</tt> loop of <tt>lnet_discover_peer_locked()</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@soak-34 ~]# nm /lib/modules/3.10.0-693.21.1.el7.x86_64/extra/lustre-client/net/lnet.ko | grep lnet_discover_peer_locked 0000000000035510 T lnet_discover_peer_locked [root@soak-34 ~]# printf '%#x\n' $((0x0000000000035510 + 0x10b)) 0x3561b [root@soak-34 ~]# addr2line -e /lib/modules/3.10.0-693.21.1.el7.x86_64/extra/lustre-client/net/lnet.ko --functions --inlines 0x3561b lnet_discover_peer_locked /usr/src/debug/lustre-2.10.58_139_g630cd49/lnet/lnet/peer.c:2064 </pre> </div></div> <p>The last lines in the debug buffer for the processing thread were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:01000000:17.0:1520891497.612056:0:13498:0:(obd_config.c:1599:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:17.0:1520891497.612058:0:13498:0:(obd_config.c:1135:class_process_config()) processing cmd: cf010 00000020:00000080:17.0:1520891497.612059:0:13498:0:(obd_config.c:1205:class_process_config()) marker 79 (0x2) soaked-OST0017 add osc </pre> </div></div> <p>It appears that the issue is that with peer discovery enabled some nodes are not responding to the ping (REPLY to the GET). Discovery is initiated in:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ptlrpc_connection_get() </pre> </div></div> <p>by calling</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LNetPrimaryNID() </pre> </div></div> <p>This function blocks until discovery completes. However because discovery is not getting a reply, the timeout for the reply is 3 minutes. In some of the logs I have seen, it shows that two OSTs are timing out, so that adds 6 minutes to mount time. What we see is that after this long period the mount completes.</p> <p>So there appears to be two problems:</p> <ol> <li>Why are some of the nodes not responding to the ping?</li> <li>The 3 minutes timeout for the ping REPLY or push ACK seems to be an overkill</li> </ol> <p>For 1, I have run a simple test where I modprobe lustre on all the nodes and then run lctl ping &lt;ost nid&gt; on all the nodes. What I found is that some would fail the ping. If I run the ping again, the nodes which failed to ping succeed the second time around. This behavior is unrelated to discovery. I'll need to look at it in more details tomorrow.</p> <p>To solve 2, there are two options to consider:</p> <ol> <li>Reduce the timeout to 5 or 10 seconds. This will reduce the time to mount the FS from 3 minutes to 10 or 20 seconds per failed server. This solution I think would be sufficient for 2.11.</li> <li>Change the behavior to do discovery in the background. Create a non-MR peer and initiate discovery on it. And at the same time continue sending messages as normal. When discovery completes the peer will be updated according to the returned reply. If discovery fails then it can be initiated again. In case of discovery failure the system operates no worse than if discovery is disabled. This can be done for 2.12</li> </ol> <p> </p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31663" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31663</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10800" title="Mount hangs on clients. " class="issue-link" data-issue-key="LU-10800"><del>LU-10800</del></a> lnet: reduce discovery timeout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3daf02b0245d57b4250e5404985b88eb72af3588</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31675" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31675</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10800" title="Mount hangs on clients. " class="issue-link" data-issue-key="LU-10800"><del>LU-10800</del></a> lnet: Revert "<a href="https://jira.whamcloud.com/browse/LU-10270" title="remove a early rx code from o2ib lnd" class="issue-link" data-issue-key="LU-10270">LU-10270</a> lnet: remove an early rx code"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46eb7f97c029a998ca384ddd208ff8444d4cb126</p> <p>John found the reason for ping being dropped. There was a patch that was introduced which removed handling the early rxs from the o2iblnd. I believe we are hitting this case. Which triggered the discovery ping timeout. We tested the revert on soak and it works properly.</p> <p>I believe we should land both the revert and the timeout change.</p> <p>The revert will eliminate the drop ping issue.</p> <p>The timeout change will deal with cases where there is a legitimate communication failure, like trying to mount while some of the OSTs are down.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31663/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31663/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10800" title="Mount hangs on clients. " class="issue-link" data-issue-key="LU-10800"><del>LU-10800</del></a> lnet: reduce discovery timeout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1cf929df259a9aaa5446a4cda637930ca5b27d7a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31675/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31675/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10800" title="Mount hangs on clients. " class="issue-link" data-issue-key="LU-10800"><del>LU-10800</del></a> lnet: Revert "<a href="https://jira.whamcloud.com/browse/LU-10270" title="remove a early rx code from o2ib lnd" class="issue-link" data-issue-key="LU-10270">LU-10270</a> lnet: remove an early rx code"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6224bb44d8d10894f1c21921a0224dd8baf0ded0</p> <p>Landed for 2.11</p> Related LU-9984 LU-10270 Development Rank 1|hzzu93: Rank (Obsolete) 9223372036854775807