https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-308 Lustre <p>On a production lustre client node, we hit an ASSERT. The first sign of trouble on the console is this:</p> <p>2011-05-11 08:55:44 LustreError: ... (mdc_locks.c:648:mdc_enqueue())<br/> ldlm_cli_enqueue: -4</p> <p>I believe that is under an emacs process.</p> <p>Ten seconds later we start getting "soft lockup" "stuck for 10s" warnings<br/> about the same process. The messages pop up every 10s until we finally get an<br/> assertion later on. Backtrace looks like:</p> <p>:mdc:mdc_enter_request<br/> :ptlrpc:ldlm_lock_addref_internal_nolock<br/> :mdc:mdc_enqueue<br/> dequeue_task<br/> thread_return<br/> :ptlrpc:ldlm_lock_add_to_lru_nolock<br/> :mdc:mdc_intent_lock<br/> :ptlrpc:ldlm_lock_decref<br/> :mdc:mdc_set_lock_data<br/> :lustre:ll_mdc_blocking_ast<br/> :ptlrpc:ldlm_completion_ast<br/> :lustre:ll_prepare_mdc_op_data<br/> :lustre:ll_lookup_it<br/> :lustre:ll_mdc_blocking_ast<br/> :lov:lov_fini_enqueue_set<br/> :lustre:ll_lookup_nd<br/> list_add<br/> d_alloc<br/> do_lookup<br/> __link_path_walk<br/> link_path_walk<br/> do_path_lookup<br/> __user_walk_fd<br/> vfs_stat_fd<br/> sys_rt_sigreturn<br/> sys_rt_sigreturn<br/> sys_newstat<br/> sys_setitimer<br/> stub_rt_sigreturn<br/> system_call</p> <p>Later a different process throws these errors:</p> <p>2011-05-11 09:06:07 Lustre: ... Request mdc_close sent 106s ago has failed due<br/> to network error (limit 106s)<br/> 2011-05-11 09:06:07 LustreError: ... ll_lcose_inode_openhandle()) inode X mdc<br/> close failed: -4<br/> 2011-05-11 09:06:07 Skipped 4 previous messages</p> <p>And then three seconds later the original stuck thread does:</p> <p>2011-05-11 09:06:10 ldlm_lock.c:189:ldlm_lock_remove_from_lru_nolock ASSERT(ns-&gt;ns_nr_unused &gt; 0) failed</p> <p>Backtrace looks like:</p> <p>ldlm_lock_remove_from_lru_nolock<br/> ldlm_lock_remove_from_lru<br/> ldlm_lock_addref_internal_nolock<br/> search_queue<br/> ldlm_lock_match<br/> ldlm_resource_get<br/> mdc_revalidate_lock<br/> ldlm_lock_addref_internal_nolock<br/> mdc_intent_lock<br/> ll_i2gids<br/> ll_prepare_mdc_op_data<br/> __ll_inode_revalidate_it<br/> ll_mdc_blocking_ast<br/> ll_inode_permission<br/> dput<br/> permission<br/> vfs_permission<br/> __link_path_walk<br/> link_path_walk<br/> do_path_lookup<br/> __path_lookup_intent_open<br/> path_lookup_open<br/> open_namei<br/> do_filp_open<br/> get_unused_fd<br/> do_sys_open<br/> sys_open</p> <p>Apologies for any typos. That all had to be hand copied.</p> <p>Since this all appears to have started with an EINTR in mdc_enqueue(), it may be that this bug is related:</p> <p> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=18213" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=18213</a><br/> <a href="http://jira.whamcloud.com/browse/LU-234" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-234</a></p> <p>We are running 1.8.5+, so we should have the fix that was applied to 1.8.5 in bug 18213.</p> RHEL5.5ish (CHAOS4.4-2), lustre 1.8.5.0-3chaos LU-308

Hang and eventual ASSERT after mdc_enqueue()) ldlm_cli_enqueue: -4

Bug Minor Resolved Fixed Lai Siyao Christopher Morrone Wed, 11 May 2011 17:37:25 +0000 Tue, 28 Jun 2011 15:01:39 +0000 Mon, 13 Jun 2011 15:00:21 +0000 Lustre 1.8.6 0 4 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Johann, could you take a look into this, I can't find a use case which will trigger ldlm_lock_remove_from_lru_nolock ASSERT(ns-&gt;ns_nr_unused &gt; 0).</p> <p>The LASSERT might just be a side effect of the initial soft lockup.<br/> We actually found &amp; fixed a problem in 1.8.5 with mdc_enter_request().<br/> See <a href="https://bugzilla.lustre.org/show_bug.cgi?id=24508#c1" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=24508#c1</a><br/> A patch was landed to Whamcloud's b1_8 as part of <a href="https://jira.whamcloud.com/browse/LU-286" title="racer: general protection fault: 0000 [1] SMP RIP: __wake_up_common+60}" class="issue-link" data-issue-key="LU-286"><del>LU-286</del></a>, see <a href="http://review.whamcloud.com/506" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/506</a></p> <p>Thank you, Johann. This looks reasonable.<br/> Chris, could you verify the patch for <a href="https://jira.whamcloud.com/browse/LU-286" title="racer: general protection fault: 0000 [1] SMP RIP: __wake_up_common+60}" class="issue-link" data-issue-key="LU-286"><del>LU-286</del></a> is not included in your test code?</p> <p>Correct, we do not have that patch.</p> <p>And our code is not "test" code; we saw this in production.</p> <p>Chris</p> <p>Will you be trying this patch in production or are some additional steps required first?</p> <p>Peter</p> <p>I'll pull it into our 1.8.5-llnl branch.</p> <p>As for when it goes in production...our local testing infrastracture has almost completely moved to RHEL6 and lustre 2.1. We have a 1.8 server cluster left over for testing, but no 1.8 clients. The first window we have to get 1.8 clients and test a release is probably mid June, with a target for installation in late June if there are no surprises.</p> <p>Let's close this ticket for now and reopen if the issue reoccurs with the patch applied</p> Development Rank 1|hzw1gv: Rank (Obsolete) 10319 Severity