https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5886 Lustre <p>During DNE II testing (same run as <a href="https://jira.whamcloud.com/browse/LU-5883" title="DNE II testing: LustreError: 6618:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed" class="issue-link" data-issue-key="LU-5883"><del>LU-5883</del></a>), both of our MDSes reported hung threads here:<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: Pid: 21117, comm: mdt02_008<br/> Nov 6 20:22:21 perses-esf-mds001 kernel:<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: Call Trace:<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0aaa1a1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e32230&gt;&#93;</span> ? ldlm_expired_completion_wait+0x0/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e36df5&gt;&#93;</span> ldlm_completion_ast+0x665/0x9a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff81065df0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e3619e&gt;&#93;</span> ldlm_cli_enqueue_local+0x21e/0x810 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e36790&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x9a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa1559d70&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x2a0 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa156240d&gt;&#93;</span> mdt_object_local_lock+0x1bd/0xa80 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa1559d70&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x2a0 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e36790&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x9a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa1562d35&gt;&#93;</span> mdt_object_lock_internal+0x65/0x360 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa15630f4&gt;&#93;</span> mdt_object_lock+0x14/0x20 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa156dd6c&gt;&#93;</span> mdt_getattr_name_lock+0xd9c/0x1a50 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff8128a0ea&gt;&#93;</span> ? strlcpy+0x4a/0x60<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e63774&gt;&#93;</span> ? lustre_msg_get_flags+0x34/0xb0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e65d40&gt;&#93;</span> ? lustre_swab_ldlm_reply+0x0/0x40 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa156ef42&gt;&#93;</span> mdt_intent_getattr+0x292/0x470 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa155bc44&gt;&#93;</span> mdt_intent_policy+0x494/0xcf0 <span class="error">&#91;mdt&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e16549&gt;&#93;</span> ldlm_lock_enqueue+0x129/0x9d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e4235b&gt;&#93;</span> ldlm_handle_enqueue0+0x51b/0x1340 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0a9a4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0ec3842&gt;&#93;</span> tgt_enqueue+0x62/0x1d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0ec40ce&gt;&#93;</span> tgt_request_handle+0x71e/0xb10 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e73964&gt;&#93;</span> ptlrpc_main+0xe64/0x1990 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff810096f0&gt;&#93;</span> ? __switch_to+0xd0/0x320<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff81527c20&gt;&#93;</span> ? thread_return+0x4e/0x76e<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffffa0e72b00&gt;&#93;</span> ? ptlrpc_main+0x0/0x1990 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff8109aee6&gt;&#93;</span> kthread+0x96/0xa0<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff8109ae50&gt;&#93;</span> ? kthread+0x0/0xa0<br/> Nov 6 20:22:21 perses-esf-mds001 kernel: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Nov 6 20:22:21 perses-esf-mds001 kernel:</p> Versions of 2.6.54 on clients &amp; servers.<br/> Cray SLES11SP3 clients, CentOS servers (2.6.32-431.5.1.el6.x86_64).<br/> <br/> Most recent commit on clients:<br/> Ie7a2a98be8cc97db9af7a64476c06fc7321544eb<br/> <a href="http://review.whamcloud.com/12142">http://review.whamcloud.com/12142</a><br/> <br/> Most recent commit on servers:<br/> If24443955290b091fd22905dfb74b0d6a6d1b4e8<br/> <a href="http://review.whamcloud.com/12490">http://review.whamcloud.com/12490</a> LU-5886

DNE II Testing: Hung threads on MDS; evictions

Bug Major Resolved Cannot Reproduce Di Wang Patrick Farrell Fri, 7 Nov 2014 20:33:14 +0000 Sun, 10 Oct 2021 22:20:22 +0000 Sun, 10 Oct 2021 22:20:22 +0000 0 4 <p>Partial Lustre logs and messages logs from both MDSes are here:<br/> ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5886" title="DNE II Testing: Hung threads on MDS; evictions" class="issue-link" data-issue-key="LU-5886"><del>LU-5886</del></a>/lustre_logs_and_messages.tar.gz</p> <p>Di,<br/> Would you be able to comment on this one?<br/> Thank you!</p> <p>According to the console log, both MDTs are waiting for the lock of a remote directory.</p> <p>mds001</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 6 20:22:21 perses-esf-mds001 kernel: Call Trace: Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0aaa1a1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e32230&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e36df5&gt;] ldlm_completion_ast+0x665/0x9a0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e3619e&gt;] ldlm_cli_enqueue_local+0x21e/0x810 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e36790&gt;] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa1559d70&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa156240d&gt;] mdt_object_local_lock+0x1bd/0xa80 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa1559d70&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e36790&gt;] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa1562d35&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa15630f4&gt;] mdt_object_lock+0x14/0x20 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa156dd6c&gt;] mdt_getattr_name_lock+0xd9c/0x1a50 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff8128a0ea&gt;] ? strlcpy+0x4a/0x60 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e63774&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e65d40&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa156ef42&gt;] mdt_intent_getattr+0x292/0x470 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa155bc44&gt;] mdt_intent_policy+0x494/0xcf0 [mdt] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e16549&gt;] ldlm_lock_enqueue+0x129/0x9d0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e4235b&gt;] ldlm_handle_enqueue0+0x51b/0x1340 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0a9a4ce&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0ec3842&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0ec40ce&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e73964&gt;] ptlrpc_main+0xe64/0x1990 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff81527c20&gt;] ? thread_return+0x4e/0x76e Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffffa0e72b00&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 Nov 6 20:22:21 perses-esf-mds001 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Nov 6 20:22:21 perses-esf-mds001 kernel: </pre> </div></div> <p>mds002</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 6 20:22:16 perses-esf-mds002 kernel: req@ffff8803330e1c00 x1484073163441132/t0(0) o36-&gt;ca4f794c-56f5-709e-3a90-0c25c515ac58@30@gni:0/0 lens 552/568 e 18 to 0 dl 1415326941 ref 2 fl Interpret:/0/0 rc 0/0 Nov 6 20:22:21 perses-esf-mds002 kernel: LNet: Service thread pid 10838 was inactive for 600.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 6 20:22:21 perses-esf-mds002 kernel: Pid: 10838, comm: mdt02_000 Nov 6 20:22:21 perses-esf-mds002 kernel: Nov 6 20:22:21 perses-esf-mds002 kernel: Call Trace: Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e01230&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e05df5&gt;] ldlm_completion_ast+0x665/0x9a0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0dffe76&gt;] ldlm_cli_enqueue_fini+0x926/0xe60 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e00761&gt;] ldlm_cli_enqueue+0x3b1/0x860 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e05790&gt;] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa1502b60&gt;] ? mdt_remote_blocking_ast+0x0/0x210 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa16d94a8&gt;] osp_md_object_lock+0x188/0x210 [osp] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa15f70bf&gt;] lod_object_lock+0x13f/0x730 [lod] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa166913b&gt;] mdd_object_lock+0x3b/0xd0 [mdd] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa150b07c&gt;] mdt_remote_object_lock+0x14c/0x320 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa150bdee&gt;] mdt_object_lock_internal+0x11e/0x360 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa150c0f4&gt;] mdt_object_lock+0x14/0x20 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa1526f80&gt;] mdt_reint_create+0x700/0xd40 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0c216b0&gt;] ? lu_ucred+0x20/0x30 [obdclass] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa1501ed5&gt;] ? mdt_ucred+0x15/0x20 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa151deec&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e57a56&gt;] ? __req_capsule_get+0x166/0x6e0 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff8128a0ea&gt;] ? strlcpy+0x4a/0x60 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa15220fd&gt;] mdt_reint_rec+0x5d/0x200 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa15061a3&gt;] mdt_reint_internal+0x4d3/0x7b0 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa1506a0b&gt;] mdt_reint+0x6b/0x120 [mdt] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e930ce&gt;] tgt_request_handle+0x71e/0xb10 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e42964&gt;] ptlrpc_main+0xe64/0x1990 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff81527c20&gt;] ? thread_return+0x4e/0x76e Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffffa0e41b00&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 Nov 6 20:22:21 perses-esf-mds002 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Unfortunately, the debug log you posted is about 1 hour earlier, could you be able to find these debug log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 6 20:22:21 perses-esf-mds002 kernel: Nov 6 20:22:21 perses-esf-mds002 kernel: LustreError: dumping log to /tmp/lustre-log.1415326941.10838 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 6 20:22:21 perses-esf-mds001 kernel: LustreError: dumping log to /tmp/lustre-log.1415326941.21117 Nov 6 20:24:27 perses-esf-mds001 kernel: Lustre: lock timed out (enqueued at 1415326767, 300s ago) </pre> </div></div> <p>Though I am not sure there are valuable information there, but worth try. Thanks.</p> <p>Patrick, could you please try current master to see if you can still reproduce the problem? Thanks.</p> Development Rank 1|hzx0i7: Rank (Obsolete) 16457 Severity