https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3919 Lustre <p>We had a deadlock recently on an MDS at NOAA. The threads were mostly waiting for a mutex in mds_lookup and for the quota master lock in target_handle_dqacq_callback, with some other ldiskfs lock waiters in there too. </p> <p>I will attach the bt, bt -f, log, and kern.log files. Let me know if there is anything else I should get from the vmcore.</p> LU-3919

deadlock on MDT, possibly related to quota?

Bug Minor Resolved Won't Fix Niu Yawei Oz Rentas Tue, 10 Sep 2013 23:07:36 +0000 Thu, 23 Jun 2016 08:12:21 +0000 Thu, 23 Jun 2016 08:12:21 +0000 Lustre 1.8.9 0 5 <p>Niu</p> <p>Could you please look into this?</p> <p>Thanks</p> <p>Peter</p> <p>Kit, that should be enough. Thanks.</p> <p>Here is the deadlock:</p> <p>A thread is holding the inode lock of _<em>iopen</em>_, then try to start journal:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 23587 TASK: ffff81082c0be860 CPU: 6 COMMAND: <span class="code-quote">"ll_mdt_29"</span> #0 [ffff8108231ad2b0] schedule at ffffffff80062fa0 #1 [ffff8108231ad388] start_this_handle at ffffffff8b269d2d #2 [ffff8108231ad408] jbd2_journal_start at ffffffff8b269ea4 #3 [ffff8108231ad428] ldiskfs_dquot_drop at ffffffff8b2aedbe #4 [ffff8108231ad448] clear_inode at ffffffff800234e7 #5 [ffff8108231ad458] dispose_list at ffffffff800352d6 #6 [ffff8108231ad488] shrink_icache_memory at ffffffff8002dcfb #7 [ffff8108231ad4c8] shrink_slab at ffffffff8003f7cd #8 [ffff8108231ad508] zone_reclaim at ffffffff800cf9ae #9 [ffff8108231ad5b8] get_page_from_freelist at ffffffff8000a8a7 #10 [ffff8108231ad628] __alloc_pages at ffffffff8000f48a #11 [ffff8108231ad698] cache_grow at ffffffff80017a52 #12 [ffff8108231ad6e8] cache_alloc_refill at ffffffff8005c3ee #13 [ffff8108231ad728] kmem_cache_alloc at ffffffff8000ac96 #14 [ffff8108231ad748] d_alloc at ffffffff80022de3 #15 [ffff8108231ad778] __lookup_hash at ffffffff80037210 #16 [ffff8108231ad7b8] lookup_one_len at ffffffff800ed617 #17 [ffff8108231ad7d8] mds_lookup at ffffffff8b367ba4 #18 [ffff8108231ad858] mds_fid2dentry at ffffffff8b359693 #19 [ffff8108231ad8f8] mds_fid2locked_dentry at ffffffff8b35b482 #20 [ffff8108231ad9b8] mds_getattr_lock at ffffffff8b35bd67 #21 [ffff8108231adb28] mds_intent_policy at ffffffff8b362453 #22 [ffff8108231adbd8] ldlm_lock_enqueue at ffffffff8aff6eb6 #23 [ffff8108231adc68] ldlm_handle_enqueue at ffffffff8b018b29 #24 [ffff8108231add08] mds_handle at ffffffff8b361210 #25 [ffff8108231ade18] ptlrpc_server_handle_request at ffffffff8b046874 #26 [ffff8108231adeb8] ptlrpc_main at ffffffff8b047f16 #27 [ffff8108231adf48] kernel_thread at ffffffff8005dfc1 </pre> </div></div> <p>Whereas, another thread started journal, then try to take inode lock of _<em>iopen</em>_:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 5862 TASK: ffff81123f6e20c0 CPU: 22 COMMAND: <span class="code-quote">"ll_mdt_rdpg_14"</span> #0 [ffff8112247dd2c0] schedule at ffffffff80062fa0 #1 [ffff8112247dd398] __mutex_lock_slowpath at ffffffff80063c63 #2 [ffff8112247dd3d8] .text.lock.mutex at ffffffff80063cad (via mutex_lock) #3 [ffff8112247dd3f8] mds_lookup at ffffffff8b367b97 #4 [ffff8112247dd478] mds_fid2dentry at ffffffff8b359693 #5 [ffff8112247dd518] mds_lvfs_fid2dentry at ffffffff8b359a5d #6 [ffff8112247dd538] llog_lvfs_create at ffffffff8af6782e #7 [ffff8112247dd5c8] llog_cat_current_log at ffffffff8af5fffa #8 [ffff8112247dd6a8] llog_cat_add_rec at ffffffff8af6217d #9 [ffff8112247dd718] llog_obd_origin_add at ffffffff8af68438 #10 [ffff8112247dd768] llog_add at ffffffff8af68cf6 #11 [ffff8112247dd7c8] lov_llog_origin_add at ffffffff8b182494 #12 [ffff8112247dd858] llog_add at ffffffff8af68cf6 #13 [ffff8112247dd8b8] mds_llog_origin_add at ffffffff8b341bc5 #14 [ffff8112247dd928] llog_add at ffffffff8af68cf6 #15 [ffff8112247dd988] mds_llog_add_unlink at ffffffff8b34173d #16 [ffff8112247dda08] mds_log_op_unlink at ffffffff8b344c8d #17 [ffff8112247dda98] mds_mfd_close at ffffffff8b384a21 #18 [ffff8112247ddbf8] mds_close at ffffffff8b38c3d0 #19 [ffff8112247ddd08] mds_handle at ffffffff8b35f67b #20 [ffff8112247dde18] ptlrpc_server_handle_request at ffffffff8b046874 #21 [ffff8112247ddeb8] ptlrpc_main at ffffffff8b047f16 #22 [ffff8112247ddf48] kernel_thread at ffffffff8005dfc1 </pre> </div></div> <p>Such kind of deadlock only happens when the memory is very tight, I can't think of a good solution for such deadlock so far. Maybe we should just return -ENOMEM and fail the getattr operation but not trigger shrinker in such case?</p> <p>Hi Niu,</p> <p>Would it be possible to return ENOMEM where the lock is held, run the shrinker outside the lock, and then retry the lookup?</p> <p>Thanks.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13743" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13743</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3919" title="deadlock on MDT, possibly related to quota?" class="issue-link" data-issue-key="LU-3919"><del>LU-3919</del></a> lvfs: fid2dentry violates locking order<br/> Project: fs/lustre-release<br/> Branch: b1_8<br/> Current Patch Set: 1<br/> Commit: 1b5c23c2ab0b2c416782e94772ff1f42817a1df3</p> <p>Close old 1.8 issue.</p> Related Development End date Thu, 23 Apr 2015 23:07:36 +0000 Rank 1|hzw1p3: Rank (Obsolete) 10356 Severity Start date Tue, 10 Sep 2013 23:07:36 +0000