https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-500 Lustre <p>At a key cutomer site we were and still are experiencing MDS thread hangs. Initially they were seen under 1.8.4 and when the MDS would dump the threads the only way to recover would be to reboot the MDS. The site did upgrade to 1.8.6 which includes a at_min patch from bug 23352 which was thought that might help the issue. However they are still seeing the thread hangs but can usually now get out of it without a MDS reboot but still a serious problem.<br/> The trace looks like:</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffff888e8c10&gt;&#93;</span> ldlm_expired_completion_wait+0x0/0x250 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888ea762&gt;&#93;</span> ldlm_completion_ast+0x4c2/0x880 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888cf709&gt;&#93;</span> ldlm_lock_enqueue+0x9d9/0xb20 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8008d07b&gt;&#93;</span> default_wake_function+0x0/0xe<br/> <span class="error">&#91;&lt;ffffffff888cab6a&gt;&#93;</span> ldlm_lock_addref_internal_nolock+0x3a/0x90 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888e92cb&gt;&#93;</span> ldlm_cli_enqueue_local+0x46b/0x520 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88becd7a&gt;&#93;</span> enqueue_ordered_locks+0x26a/0x4d0 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888e6bc0&gt;&#93;</span> ldlm_blocking_ast+0x0/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888ea2a0&gt;&#93;</span> ldlm_completion_ast+0x0/0x880 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88bed5c1&gt;&#93;</span> mds_get_parent_child_locked+0x5e1/0x8a0 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88c0f394&gt;&#93;</span> mds_open+0xc44/0x35f8 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8899c2b6&gt;&#93;</span> kiblnd_post_tx_locked+0x566/0x730 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888e6d5e&gt;&#93;</span> ldlm_blocking_ast+0x19e/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff887dcb38&gt;&#93;</span> upcall_cache_get_entry+0x958/0xa50 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888eb9b8&gt;&#93;</span> ldlm_handle_bl_callback+0x1c8/0x230 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88be7f49&gt;&#93;</span> mds_reint_rec+0x1d9/0x2b0 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88c13c32&gt;&#93;</span> mds_open_unpack+0x312/0x430 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88bdae7a&gt;&#93;</span> mds_reint+0x35a/0x420 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88bd9d8a&gt;&#93;</span> fixup_handle_for_resent_req+0x5a/0x2c0 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88be4bfc&gt;&#93;</span> mds_intent_policy+0x4ac/0xc80 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888d18b6&gt;&#93;</span> ldlm_resource_putref+0x1b6/0x3c0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888ceeb6&gt;&#93;</span> ldlm_lock_enqueue+0x186/0xb20 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888cb7fd&gt;&#93;</span> ldlm_lock_create+0x9bd/0x9f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888f3720&gt;&#93;</span> ldlm_server_blocking_ast+0x0/0x83d <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff888f0849&gt;&#93;</span> ldlm_handle_enqueue+0xbf9/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88be3b20&gt;&#93;</span> mds_handle+0x4130/0x4d60 <span class="error">&#91;mds&#93;</span><br/> <span class="error">&#91;&lt;ffffffff887ffbe5&gt;&#93;</span> lnet_match_blocked_msg+0x375/0x390 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;&lt;ffffffff88914705&gt;&#93;</span> lustre_msg_get_conn_cnt+0x35/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8006e244&gt;&#93;</span> do_gettimeoffset_tsc+0x19/0x3c<br/> <span class="error">&#91;&lt;ffffffff8891bc37&gt;&#93;</span> ptlrpc_server_handle_request+0xaa7/0x1150 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8008ca80&gt;&#93;</span> __activate_task+0x56/0x6d<br/> <span class="error">&#91;&lt;ffffffff8008c865&gt;&#93;</span> dequeue_task+0x18/0x37<br/> <span class="error">&#91;&lt;ffffffff80062ff8&gt;&#93;</span> thread_return+0x62/0xfe<br/> <span class="error">&#91;&lt;ffffffff8003da91&gt;&#93;</span> lock_timer_base+0x1b/0x3c<br/> <span class="error">&#91;&lt;ffffffff8001cb46&gt;&#93;</span> __mod_timer+0x100/0x10f<br/> <span class="error">&#91;&lt;ffffffff8891f698&gt;&#93;</span> ptlrpc_main+0x1258/0x1420 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8008d07b&gt;&#93;</span> default_wake_function+0x0/0xe<br/> <span class="error">&#91;&lt;ffffffff800b7a9c&gt;&#93;</span> audit_syscall_exit+0x336/0x362<br/> <span class="error">&#91;&lt;ffffffff8005dfb1&gt;&#93;</span> child_rip+0xa/0x11<br/> <span class="error">&#91;&lt;ffffffff8891e440&gt;&#93;</span> ptlrpc_main+0x0/0x1420 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8005dfa7&gt;&#93;</span> child_rip+0x0/0x11</p> CentOS 5.3 LU-500

MDS threads hang ldlm_expired_completion_wait+

Bug Major Resolved Duplicate Oleg Drokin Steven Woods Tue, 12 Jul 2011 17:15:04 +0000 Thu, 29 Mar 2012 11:04:23 +0000 Thu, 29 Mar 2012 11:04:23 +0000 Lustre 1.8.6 0 12 <p>Steve</p> <p>Could you please clarify as to which site(s) is\are affected? I am finding the history in the Oracle bz ticket confusing.</p> <p>Thanks</p> <p>Peter</p> <p>Sorry - ORNL spider</p> <p>Oleg</p> <p>Could you please look at this one?</p> <p>Thanks</p> <p>Peter</p> <p>Oleg,</p> <p>You mentioned that you were going to talk to ORNL about this ticket. Could you please provide a status update?</p> <p>Thanks</p> <p>Peter</p> <p>So how do you get out of the thread hungs now without a reboot?<br/> Just waiting it out and eventually whatever client was unresponsive was evicted? If so that's the way things are, we must give a chance to the client to respond, if it cannot respond in time for whatever reason, we have no way of knowing unless the timeout is reached.</p> <p>Can you please share the logs from 1.8.6?</p> <p>Attached 1.8.6 traces</p> <p>Oleg, Steve can answer questions about the most recent dumps. Internally, we've seen similar behavior on internal Cray systems. In those cases we see that an unresponsive client does eventually get evicted and the MDS recovers, but the service threads that are waiting for the blocking callbacks to complete never seem to unwedge. Also in these cases, the client initially fails for something completely unrelated; e.g. kernel null dereference. ...not sure that it is the same symptoms witnessed in the Spider dumps.</p> <p>AFAIK initially before the patch to 23352 things never cleared up until a reboot. After that they can keep running but I do not believe the thread ever recovers.</p> <p>If the thread never recovers I am interested to see a backtrace dump from such a thread.<br/> Unfortunately the kernel log excerpt is way to small. We see a few backtraces from threads blocked, then a client is evicted an that's where the log ends. So I have no idea if the thread is doing anything else reports anything else and so on.</p> <p>I am nor sure how rpctrace is going to help, though.</p> <p>It seems that the issue is common to 1.8.6 and 2.1.</p> <p>I've found our MDS (2.1) hanging today morning with plenty of the same errors that Steven Reported.<br/> Our Environment consists of 1.8.6-wc1 clients and 2.1 servers.</p> <p>In order to recover from it we have remounted MDT resource. And recovery begun. Unfrotunatelly we have never seen <br/> successfull MDS recovery in 2.1. This time system was able to recover 460/890 clients and got stuck (transaction numbers were increasing until the timeout occured, and one client got evicted, this loop could take a long time so we had to abort it)</p> <p>We can provide lustre log files and kernel stack traces if needed.</p> <p>Lukasz, wcan you share exact backtrace? Also how long were yo waiting to relieve the situation?<br/> Usually such things are happening due to a packet loss (or a node death at a bad time).</p> <p>Additionally if you are having constant recovery problems, can you please file a separate ticket for it?</p> <p>Thanks.</p> <p>I believe that the 1.8.6 instances of this bug are caused by incorrect lock ordering introduced by the patch from bug 24437 and is being pursued under <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a>.</p> <p>Can this bug be closed now?</p> <p>on 2/Sep/11 I mentioned <a href="https://jira.whamcloud.com/browse/LU-146" title="executable files on NFS share failed with &quot;Text file busy&quot; when executed" class="issue-link" data-issue-key="LU-146"><del>LU-146</del></a>. Although that could have been a contributor, the serialization of ptlrpc sets of size PARALLEL_AST_LIMIT cause these threads to trigger watchdog. I think we need the patches posted to bz 24450 pulled to WC's b1_8 in order to close this ticket.</p> <p>Which patches from bz 24450?</p> <p>&gt; Which patches from bz 24450?<br/> The ones that Oracle has landed; namely attachments 33099, 33106, 33137, and 33144.</p> <p><a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a> has those patches ported to Lustre 1.8-wc branch. If those patches are the solution then this ticket can be marked as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a>. What do you say Cory?</p> <p>James, that sounds right to me. This should now be closed as a dup of <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a>.</p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a></p> Bugzilla ID 24450.0 Development Rank 1|hzvi3z: Rank (Obsolete) 6583 Severity