https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5497 Lustre <p>Our production MDS systems occasionally get stuck with many service threads stuck in <tt>ldlm_completion_ast()</tt>. The details were described in <a href="https://jira.whamcloud.com/browse/LU-4579" title="Timeout system horribly broken" class="issue-link" data-issue-key="LU-4579"><del>LU-4579</del></a>, but that issue was closed when the patch landed which fixed how timeouts are reported.</p> <p>When this happens, client access hangs and the MDS appears completely idle.</p> LU-5497

Many MDS service threads blocked in ldlm_completion_ast()

Bug Blocker Closed Won't Fix Oleg Drokin Ned Bass llnl Fri, 15 Aug 2014 20:34:48 +0000 Sun, 14 Aug 2016 17:28:30 +0000 Sun, 14 Aug 2016 17:28:30 +0000 Lustre 2.4.2 0 13 <p>Example stack trace.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-02-03 11:49:01 LustreError: dumping log to /tmp/lustre-log.1391456941.15242 2014-02-03 11:49:02 Pid: 14993, comm: mdt00_011 2014-02-03 11:49:02 2014-02-03 11:49:02 Call Trace: 2014-02-03 11:49:02 [&lt;ffffffffa05cc341&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2014-02-03 11:49:02 [&lt;ffffffffa05bc77e&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] 2014-02-03 11:49:02 [&lt;ffffffffa08914ca&gt;] ldlm_completion_ast+0x57a/0x960 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa088cb60&gt;] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffff81063ba0&gt;] ? default_wake_function+0x0/0x20 2014-02-03 11:49:02 [&lt;ffffffffa0890b78&gt;] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa0890f50&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa0e42a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e48c7b&gt;] mdt_object_lock0+0x33b/0xaf0 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e42a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0890f50&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa0e494f4&gt;] mdt_object_lock+0x14/0x20 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e586d9&gt;] mdt_getattr_name_lock+0xe09/0x1960 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa08b9cf5&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa08e2846&gt;] ? __req_capsule_get+0x166/0x700 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa08bbf84&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa0e594cd&gt;] mdt_intent_getattr+0x29d/0x490 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e45f1e&gt;] mdt_intent_policy+0x39e/0x720 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa08718b1&gt;] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa089a9df&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa0e463a6&gt;] mdt_enqueue+0x46/0xe0 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e4d758&gt;] mdt_handle_common+0x648/0x1660 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa0e86405&gt;] mds_regular_handle+0x15/0x20 [mdt] 2014-02-03 11:49:02 [&lt;ffffffffa08caf88&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa05bc63e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 2014-02-03 11:49:02 [&lt;ffffffffa05cde0f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 2014-02-03 11:49:02 [&lt;ffffffffa08c22e9&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa08cc31e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa08cb850&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 2014-02-03 11:49:02 [&lt;ffffffffa08cb850&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffffa08cb850&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 2014-02-03 11:49:02 [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 2014-02-03 11:49:02 </pre> </div></div> <p>Oleg's initial reaction to this ticket was that this was <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> (which you are evaluating a fix for).</p> <p>Oleg would you care to elaborate?</p> <p>This looks like one of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> fallouts.</p> <p>Is Livermore running 2.4 with <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> or <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a>?</p> <p>We are using <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> with 2.4.</p> <p>Was this with lu4584 in the original form?<br/> There's not too much data here, but I observed similar lockups in my testing using your chaos tree initially.</p> <p>Using <a href="http://review.whamcloud.com/#/c/6511/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6511/</a> + the latest form of lu4584 patch has a high chance of eliminating this problem too, I think.</p> <p>Oleg - <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> is... long. Is the patch you're referring to this one: <a href="http://review.whamcloud.com/#/c/9488/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9488/</a> ?</p> <p>Also, Chris, James, et al. - Any further updates on this? Have the patches been tried? Are you still seeing the problems? Etc.</p> <p>We have avoided this problem by going to 2.5 for most of our systems. The patch <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> should help with this but if I remember right Chris reported issues at <a href="https://jira.whamcloud.com/browse/LU-5525" title="ASSERTION( new_lock-&gt;l_readers + new_lock-&gt;l_writers == 0 ) failed" class="issue-link" data-issue-key="LU-5525"><del>LU-5525</del></a> due to that patch. </p> <p>We ran Patch Set 9 of <a href="http://review.whamcloud.com/9488/" class="external-link" target="_blank" rel="nofollow noopener">change 9488</a> as part of 2.4.2, but we dropped it in favor of whatever landed on b2_5 now that we are based on 2.5.3.</p> <p>Looks like we have this this issue at 3 times in the past 48 hours. Lots of threads stuck at ldlm_completion_ast. We are running with 2.4.3. It is not clear from the conversations above what patch is need.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LNet: Service thread pid 4390 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Pid: 4390, comm: mdt00_094 Call Trace: [&lt;ffffffffa07a58b5&gt;] ? _ldlm_lock_debug+0x2d5/0x660 [ptlrpc] [&lt;ffffffff815404c2&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81080610&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa050d6d1&gt;] cfs_waitq_timedwait+0x11/0x20 [libcfs] [&lt;ffffffffa07c9fed&gt;] ldlm_completion_ast+0x4ed/0x960 [ptlrpc] [&lt;ffffffffa07c5780&gt;] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc] [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa07c9728&gt;] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] [&lt;ffffffffa07c9b00&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] [&lt;ffffffffa0da1a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] [&lt;ffffffffa0da7cbb&gt;] mdt_object_lock0+0x33b/0xaf0 [mdt] [&lt;ffffffffa0da1a90&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] [&lt;ffffffffa07c9b00&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] [&lt;ffffffffa0da8534&gt;] mdt_object_lock+0x14/0x20 [mdt] [&lt;ffffffffa0db77a9&gt;] mdt_getattr_name_lock+0xe19/0x1980 [mdt] [&lt;ffffffffa07f2125&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] [&lt;ffffffffa081a636&gt;] ? __req_capsule_get+0x166/0x700 [ptlrpc] [&lt;ffffffffa07f43b4&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] [&lt;ffffffffa0db85b2&gt;] mdt_intent_getattr+0x2a2/0x4b0 [mdt] [&lt;ffffffffa0da4f3e&gt;] mdt_intent_policy+0x39e/0x720 [mdt] [&lt;ffffffffa07aa831&gt;] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] [&lt;ffffffffa07d11bf&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] [&lt;ffffffffa0da53c6&gt;] mdt_enqueue+0x46/0xe0 [mdt] [&lt;ffffffffa0dabad7&gt;] mdt_handle_common+0x647/0x16d0 [mdt] [&lt;ffffffffa0de58f5&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa08033b8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [&lt;ffffffffa050d5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa051ed5f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa07fa719&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [&lt;ffffffff81055813&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa080474e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] [&lt;ffffffffa0803c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa0803c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffffa0803c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>We have faced this issue in 2.1.0 build. There are too many hung threads in <b>ldlm_completion_ast</b>. Is there any work around for this ? </p> <p>Oleg, can you please point to the patch which fixes this problem ?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 240529 TASK: ffff8810288aab40 CPU: 8 COMMAND: "mdt_46" #0 [ffff880fc6d1b860] schedule at ffffffff814ea122 #1 [ffff880fc6d1b928] cfs_waitq_wait at ffffffffa041c69e [libcfs] #2 [ffff880fc6d1b938] ldlm_completion_ast at ffffffffa0709722 [ptlrpc] #3 [ffff880fc6d1b9e8] ldlm_cli_enqueue_local at ffffffffa0708e11 [ptlrpc] #4 [ffff880fc6d1ba78] mdt_object_lock at ffffffffa0bf2a8d [mdt] #5 [ffff880fc6d1bb18] mdt_getattr_name_lock at ffffffffa0bff710 [mdt] #6 [ffff880fc6d1bbb8] mdt_intent_getattr at ffffffffa0c005cd [mdt] #7 [ffff880fc6d1bc08] mdt_intent_policy at ffffffffa0c01679 [mdt] #8 [ffff880fc6d1bc48] ldlm_lock_enqueue at ffffffffa06ea1f9 [ptlrpc] #9 [ffff880fc6d1bca8] ldlm_handle_enqueue0 at ffffffffa07120ef [ptlrpc] #10 [ffff880fc6d1bd18] mdt_enqueue at ffffffffa0c01a16 [mdt] #11 [ffff880fc6d1bd38] mdt_handle_common at ffffffffa0bf4ffa [mdt] #12 [ffff880fc6d1bd88] mdt_regular_handle at ffffffffa0bf5eb5 [mdt] #13 [ffff880fc6d1bd98] ptlrpc_main at ffffffffa0742e0e [ptlrpc] #14 [ffff880fc6d1bee8] kthread at ffffffff81090806 #15 [ffff880fc6d1bf48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>Old blocker for unsupported version</p> Related LU-4579 Development End date Wed, 13 Jan 2016 20:34:48 +0000 Rank 1|hzwtx3: Rank (Obsolete) 15338 Severity Start date Fri, 15 Aug 2014 20:34:48 +0000