https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17103 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5c0452f5-d96f-4c8d-9d27-04b71f59c2ee" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5c0452f5-d96f-4c8d-9d27-04b71f59c2ee</a></p> <p>test_207 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 400 minutes, last suite running was sanity-lnet </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/96852" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/96852</a> - 4.18.0-425.10.1.el8_7.aarch64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/96852" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/96852</a> - 4.18.0-477.15.1.el8_lustre.x86_64</p> <p>It looks like this started failing on 2023-08-11 before patch <a href="https://review.whamcloud.com/51921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/51921</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-17006" title="socklnd: modify ksocklnd-config " class="issue-link" data-issue-key="LU-17006"><del>LU-17006</del></a> lnet: set up routes for going across subnets</tt>" landed, then started failing for everyone when it landed to master on 2023-08-31.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lnet test_207 - Timeout occurred after 400 minutes, last suite running was sanity-lnet</p> LU-17103

sanity-lnet test_207: timed out

Bug Critical Resolved Fixed Serguei Smirnov Maloo Sat, 9 Sep 2023 00:49:35 +0000 Thu, 9 Nov 2023 08:39:33 +0000 Wed, 8 Nov 2023 22:42:32 +0000 Lustre 2.16.0 0 10 <p>Serguei, could you please investigate/reassign. It looks like this is a test regression from landing your routes across subnets patch.</p> <p><a href="https://testing.whamcloud.com/search?horizon=7776000&amp;status%5B%5D=TIMEOUT&amp;test_set_script_id=a2b1c4b2-b449-11e9-b88c-52540065bddc&amp;sub_test_script_id=2df9a444-4299-4232-9b37-fc5fe71b6614&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?horizon=7776000&amp;status%5B%5D=TIMEOUT&amp;test_set_script_id=a2b1c4b2-b449-11e9-b88c-52540065bddc&amp;sub_test_script_id=2df9a444-4299-4232-9b37-fc5fe71b6614&amp;source=sub_tests#redirect</a></p> <p>"Serguei Smirnov &lt;ssmirnov@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52341</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17103" title="sanity-lnet test_207: timed out" class="issue-link" data-issue-key="LU-17103"><del>LU-17103</del></a> socklnd: revert <a href="https://jira.whamcloud.com/browse/LU-17006" title="socklnd: modify ksocklnd-config " class="issue-link" data-issue-key="LU-17006"><del>LU-17006</del></a> routing setup change<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ab2b4363141dd11d3dd6e16c4076afce359dbb77</p> <p>So far all failed cases have similar trace:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [Tue Sep 19 20:54:39 2023] task:lnetctl         state:D stack:    0 pid:949425 ppid:948947 flags:0x00004080 [Tue Sep 19 20:54:39 2023] Call Trace: [Tue Sep 19 20:54:39 2023]  __schedule+0x2d1/0x870 [Tue Sep 19 20:54:39 2023]  schedule+0x55/0xf0 [Tue Sep 19 20:54:39 2023]  schedule_timeout+0x197/0x300 [Tue Sep 19 20:54:39 2023]  ? __next_timer_interrupt+0xf0/0xf0 [Tue Sep 19 20:54:39 2023]  ? prepare_to_wait_event+0xa0/0x180 [Tue Sep 19 20:54:39 2023]  lnet_ping_md_unlink+0xd0/0x190 [lnet] [Tue Sep 19 20:54:39 2023]  ? init_wait_var_entry+0x50/0x50 [Tue Sep 19 20:54:39 2023]  lnet_ping_target_update+0x490/0x580 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_add_net_common+0x28b/0x490 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_dyn_add_ni+0x24c/0x2f0 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_genl_parse_local_ni.isra.47+0x190/0x1180 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_net_cmd+0x2fe/0x6e0 [lnet] [Tue Sep 19 20:54:39 2023]  genl_family_rcv_msg_doit.isra.17+0x113/0x150 [Tue Sep 19 20:54:39 2023]  genl_family_rcv_msg+0xb7/0x170 [Tue Sep 19 20:54:39 2023]  ? lnet_dyn_del_net+0x1f0/0x1f0 [lnet] [Tue Sep 19 20:54:39 2023]  ? lnet_shutdown_lndnets+0x300/0x300 [lnet] [Tue Sep 19 20:54:39 2023]  ? lnet_counters_get+0x140/0x140 [lnet] [Tue Sep 19 20:54:39 2023]  ? lnet_ping_event_handler+0x130/0x130 [lnet] [Tue Sep 19 20:54:39 2023]  genl_rcv_msg+0x47/0xa0 [Tue Sep 19 20:54:39 2023]  ? genl_family_rcv_msg+0x170/0x170 [Tue Sep 19 20:54:39 2023]  netlink_rcv_skb+0x4c/0x130 [Tue Sep 19 20:54:39 2023]  genl_rcv+0x24/0x40 [Tue Sep 19 20:54:39 2023]  netlink_unicast+0x19a/0x230 [Tue Sep 19 20:54:39 2023]  netlink_sendmsg+0x204/0x3d0 [Tue Sep 19 20:54:39 2023]  sock_sendmsg+0x50/0x60 [Tue Sep 19 20:54:39 2023]  ____sys_sendmsg+0x22a/0x250 [Tue Sep 19 20:54:39 2023]  ? copy_msghdr_from_user+0x5c/0x90 [Tue Sep 19 20:54:39 2023]  ? ____sys_recvmsg+0xb0/0x150 [Tue Sep 19 20:54:39 2023]  ___sys_sendmsg+0x7c/0xc0 [Tue Sep 19 20:54:39 2023]  ? copy_msghdr_from_user+0x5c/0x90 [Tue Sep 19 20:54:39 2023]  ? ___sys_recvmsg+0x89/0xc0 [Tue Sep 19 20:54:39 2023]  ? __wake_up_common_lock+0x89/0xc0 [Tue Sep 19 20:54:39 2023]  __sys_sendmsg+0x57/0xa0 [Tue Sep 19 20:54:39 2023]  do_syscall_64+0x5b/0x1b0 [Tue Sep 19 20:54:39 2023]  entry_SYSCALL_64_after_hwframe+0x61/0xc6 [Tue Sep 19 20:54:39 2023] task:monitor_thread  state:D stack:    0 pid:16801 ppid:     2 flags:0x80004080 [Tue Sep 19 20:54:39 2023] Call Trace: [Tue Sep 19 20:54:39 2023]  __schedule+0x2d1/0x870 [Tue Sep 19 20:54:39 2023]  schedule+0x55/0xf0 [Tue Sep 19 20:54:39 2023]  schedule_preempt_disabled+0xa/0x10 [Tue Sep 19 20:54:39 2023]  __mutex_lock.isra.7+0x349/0x420 [Tue Sep 19 20:54:39 2023]  ? vsnprintf+0x380/0x520 [Tue Sep 19 20:54:39 2023]  lnet_peerni_by_nid_locked+0x5f/0x100 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_select_pathway+0xd7/0x1ae0 [lnet] [Tue Sep 19 20:54:39 2023]  ? cfs_trace_unlock_tcd+0x20/0x70 [libcfs] [Tue Sep 19 20:54:39 2023]  ? libcfs_debug_msg+0x98d/0xb10 [libcfs] [Tue Sep 19 20:54:39 2023]  lnet_send+0x6d/0x1e0 [lnet] [Tue Sep 19 20:54:39 2023]  LNetGet+0x384/0x880 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_send_ping+0x116/0x1f0 [lnet] [Tue Sep 19 20:54:39 2023]  ? lnet_monitor_thread+0x670/0x670 [lnet] [Tue Sep 19 20:54:39 2023]  ? lnet_monitor_thread+0x670/0x670 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_recover_local_nis.constprop.40+0x326/0x740 [lnet] [Tue Sep 19 20:54:39 2023]  lnet_monitor_thread+0x3b0/0x670 [lnet] [Tue Sep 19 20:54:39 2023]  ? update_load_avg+0x7e/0x710 [Tue Sep 19 20:54:39 2023]  ? update_load_avg+0x7e/0x710 [Tue Sep 19 20:54:39 2023]  ? set_next_entity+0xb5/0x1e0 [Tue Sep 19 20:54:39 2023]  ? __raw_spin_unlock_irq+0x5/0x20 [Tue Sep 19 20:54:39 2023]  ? finish_task_switch+0x86/0x2e0 [Tue Sep 19 20:54:39 2023]  ? __schedule+0x2d9/0x870 [Tue Sep 19 20:54:39 2023]  ? lnet_recover_local_nis.constprop.40+0x740/0x740 [lnet] [Tue Sep 19 20:54:39 2023]  kthread+0x134/0x150 [Tue Sep 19 20:54:39 2023]  ? set_kthread_struct+0x50/0x50 [Tue Sep 19 20:54:39 2023]  ret_from_fork+0x35/0x40</pre> </div></div> <p> </p> <p>From <a href="https://testing.whamcloud.com/test_logs/d7f79d84-b2fa-4a99-95cd-2d55f67fa07c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/d7f79d84-b2fa-4a99-95cd-2d55f67fa07c/show_text</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:00000200:1.0:1693543306.254808:1168:7233:0:(peer.c:3885:lnet_peer_send_push()) peer 10.240.44.204@tcp 00000400:00000200:1.0:1693543306.254811:992:7233:0:(peer.c:4085:lnet_peer_discovery()) peer 10.240.44.204@tcp(00000000e0dde996) state 0x571 rc 0 00000800:00000100:0.0:1693543356.094024:1280:7229:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 48 0@&lt;0:0&gt;-&gt;10.240.44.204@tcp 00000400:00000200:0.0:1693543356.094053:1664:7229:0:(lib-msg.c:819:lnet_health_check()) health check: 10.240.44.209@tcp-&gt;10.240.44.204@tcp: PUT: LOCAL_TIMEOUT 00000400:00000200:0.0:1693543356.094078:1792:7229:0:(lib-msg.c:620:lnet_resend_msg_locked()) 10.240.44.209@tcp-&gt;10.240.44.204@tcp:PUT:LOCAL_TIMEOUT - queuing msg (000000007e1d92a1) for resend 00000400:00000200:1.0:1693543356.094161:1744:7234:0:(lib-move.c:3437:lnet_resend_pending_msgs_locked()) resending &lt;?&gt;-&gt;12345-10.240.44.204@tcp: PUT recovery 0 try# 1 00000400:00000200:1.0:1693543356.094182:2288:7234:0:(lib-move.c:2945:lnet_handle_send_case_locked()) Source ANY to MR: 10.240.44.204@tcp local destination 00000400:00000200:1.0:1693543356.094214:2496:7234:0:(lib-move.c:1422:lnet_select_peer_ni()) sd_best_lpni = 10.240.44.204@tcp 00000400:00000200:1.0:1693543356.094222:2448:7234:0:(lib-move.c:1895:lnet_handle_send()) 10.240.44.209@tcp NI seq info: [98481:98481:256:4294967295] 10.240.44.204@tcp LPNI seq info [10:10:8:4294967295] 00000400:00000200:1.0:1693543356.094234:2448:7234:0:(lib-move.c:1997:lnet_handle_send()) rspt_next_hop_nid = 10.240.44.204@tcp 00000400:00000200:1.0:1693543356.094244:2448:7234:0:(lib-move.c:2012:lnet_handle_send()) TRACE: 10.240.44.209@tcp(10.240.44.209@tcp:&lt;?&gt;) -&gt; 10.240.44.204@tcp(10.240.44.204@tcp:10.240.44.204@tcp) &lt;?&gt; : PUT try# 1 00000800:00000200:1.0:1693543356.094252:1952:7234:0:(socklnd_cb.c:1007:ksocknal_send()) sending 48 bytes in 1 frags to 12345-10.240.44.204@tcp 00000800:00000200:1.0:1693543356.094270:2304:7234:0:(socklnd.c:246:ksocknal_find_peer_locked()) got peer_ni [00000000d7cebc4a] -&gt; 12345-10.240.44.204@tcp (2) 00000800:00000200:1.1:1693543356.094277:2304:7234:0:(socklnd.c:246:ksocknal_find_peer_locked()) got peer_ni [00000000d7cebc4a] -&gt; 12345-10.240.44.204@tcp (2) 00000400:00000100:1.0:1693543380.174252:1376:7224:0:(acceptor.c:109:lnet_connect_console_error()) Connection to 10.240.44.204@tcp at host 10.240.44.204:7988 took too long: that node may be hung or experiencing high load. 00000400:00000200:1.0:1693543380.174267:1344:7224:0:(router.c:1691:lnet_notify()) 10.240.44.209@tcp notifying 10.240.44.204@tcp: down 00000800:00000100:1.0:1693543380.174288:1264:7224:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 48 0@&lt;0:0&gt;-&gt;10.240.44.204@tcp 00000400:00000200:1.0:1693543380.174311:1648:7224:0:(lib-msg.c:819:lnet_health_check()) health check: 10.240.44.209@tcp-&gt;10.240.44.204@tcp: PUT: LOCAL_TIMEOUT 00000400:00000200:1.0:1693543380.174332:1776:7224:0:(lib-msg.c:620:lnet_resend_msg_locked()) 10.240.44.209@tcp-&gt;10.240.44.204@tcp:PUT:LOCAL_TIMEOUT - queuing msg (000000007e1d92a1) for resend </pre> </div></div> <p>The PUSH is on resendq, but monitor thread is stuck trying to get mutex. lnetctl PID holding mutex is waiting for this PUSH to be finalized. Deadlock</p> <p>"Chris Horn &lt;chris.horn@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52479" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52479</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17103" title="sanity-lnet test_207: timed out" class="issue-link" data-issue-key="LU-17103"><del>LU-17103</del></a> lnet: Avoid deadlock when updating ping buffer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6ad2cc51c56b8d0b4745ef585a10665b925ef0f6</p> <p>"Serguei Smirnov &lt;ssmirnov@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52522" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52522</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17103" title="sanity-lnet test_207: timed out" class="issue-link" data-issue-key="LU-17103"><del>LU-17103</del></a> lnet: use workqueue for lnd ping buffer updates<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d95f9f0ff460f8ca794d8cadf4b481afce881fe8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52479/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52479/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17103" title="sanity-lnet test_207: timed out" class="issue-link" data-issue-key="LU-17103"><del>LU-17103</del></a> lnet: Avoid deadlock when updating ping buffer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3ca6ba39a21cfebc81bbe7f889c486bb82bb563a</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52522/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52522/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17103" title="sanity-lnet test_207: timed out" class="issue-link" data-issue-key="LU-17103"><del>LU-17103</del></a> lnet: use workqueue for lnd ping buffer updates<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a8e66b899afc0bc09285d426e3acef9d3d1f7694</p> <p>Merged for 2.16</p> Duplicate Related LU-17207 Development Rank 1|i03v3z: Rank (Obsolete) 9223372036854775807 Severity