[LU-3919] deadlock on MDT, possibly related to quota?

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.9
• Type: Bug
• Priority: Minor
• Reporter: Oz Rentas
• Assignee: Niu Yawei (Inactive)
• Resolution: Won't Fix
• Severity: 3

Description

We had a deadlock recently on an MDS at NOAA. The threads were mostly waiting for a mutex in mds_lookup and for the quota master lock in target_handle_dqacq_callback, with some other ldiskfs lock waiters in there too.

I will attach the bt, bt -f, log, and kern.log files. Let me know if there is anything else I should get from the vmcore.

Comments

[Peter Jones]

Niu

Could you please look into this?

Thanks

Peter

[Niu Yawei (Inactive)]

Kit, that should be enough. Thanks.

[Niu Yawei (Inactive)]

Here is the deadlock:

A thread is holding the inode lock of _iopen_, then try to start journal:

PID: 23587  TASK: ffff81082c0be860  CPU: 6   COMMAND: "ll_mdt_29"
 #0 [ffff8108231ad2b0] schedule at ffffffff80062fa0
 #1 [ffff8108231ad388] start_this_handle at ffffffff8b269d2d
 #2 [ffff8108231ad408] jbd2_journal_start at ffffffff8b269ea4
 #3 [ffff8108231ad428] ldiskfs_dquot_drop at ffffffff8b2aedbe
 #4 [ffff8108231ad448] clear_inode at ffffffff800234e7
 #5 [ffff8108231ad458] dispose_list at ffffffff800352d6
 #6 [ffff8108231ad488] shrink_icache_memory at ffffffff8002dcfb
 #7 [ffff8108231ad4c8] shrink_slab at ffffffff8003f7cd
 #8 [ffff8108231ad508] zone_reclaim at ffffffff800cf9ae
 #9 [ffff8108231ad5b8] get_page_from_freelist at ffffffff8000a8a7
#10 [ffff8108231ad628] __alloc_pages at ffffffff8000f48a
#11 [ffff8108231ad698] cache_grow at ffffffff80017a52
#12 [ffff8108231ad6e8] cache_alloc_refill at ffffffff8005c3ee
#13 [ffff8108231ad728] kmem_cache_alloc at ffffffff8000ac96
#14 [ffff8108231ad748] d_alloc at ffffffff80022de3
#15 [ffff8108231ad778] __lookup_hash at ffffffff80037210
#16 [ffff8108231ad7b8] lookup_one_len at ffffffff800ed617
#17 [ffff8108231ad7d8] mds_lookup at ffffffff8b367ba4
#18 [ffff8108231ad858] mds_fid2dentry at ffffffff8b359693
#19 [ffff8108231ad8f8] mds_fid2locked_dentry at ffffffff8b35b482
#20 [ffff8108231ad9b8] mds_getattr_lock at ffffffff8b35bd67
#21 [ffff8108231adb28] mds_intent_policy at ffffffff8b362453
#22 [ffff8108231adbd8] ldlm_lock_enqueue at ffffffff8aff6eb6
#23 [ffff8108231adc68] ldlm_handle_enqueue at ffffffff8b018b29
#24 [ffff8108231add08] mds_handle at ffffffff8b361210
#25 [ffff8108231ade18] ptlrpc_server_handle_request at ffffffff8b046874
#26 [ffff8108231adeb8] ptlrpc_main at ffffffff8b047f16
#27 [ffff8108231adf48] kernel_thread at ffffffff8005dfc1

Whereas, another thread started journal, then try to take inode lock of _iopen_:

PID: 5862   TASK: ffff81123f6e20c0  CPU: 22  COMMAND: "ll_mdt_rdpg_14"
 #0 [ffff8112247dd2c0] schedule at ffffffff80062fa0
 #1 [ffff8112247dd398] __mutex_lock_slowpath at ffffffff80063c63
 #2 [ffff8112247dd3d8] .text.lock.mutex at ffffffff80063cad (via mutex_lock)
 #3 [ffff8112247dd3f8] mds_lookup at ffffffff8b367b97
 #4 [ffff8112247dd478] mds_fid2dentry at ffffffff8b359693
 #5 [ffff8112247dd518] mds_lvfs_fid2dentry at ffffffff8b359a5d
 #6 [ffff8112247dd538] llog_lvfs_create at ffffffff8af6782e
 #7 [ffff8112247dd5c8] llog_cat_current_log at ffffffff8af5fffa
 #8 [ffff8112247dd6a8] llog_cat_add_rec at ffffffff8af6217d
 #9 [ffff8112247dd718] llog_obd_origin_add at ffffffff8af68438
#10 [ffff8112247dd768] llog_add at ffffffff8af68cf6
#11 [ffff8112247dd7c8] lov_llog_origin_add at ffffffff8b182494
#12 [ffff8112247dd858] llog_add at ffffffff8af68cf6
#13 [ffff8112247dd8b8] mds_llog_origin_add at ffffffff8b341bc5
#14 [ffff8112247dd928] llog_add at ffffffff8af68cf6
#15 [ffff8112247dd988] mds_llog_add_unlink at ffffffff8b34173d
#16 [ffff8112247dda08] mds_log_op_unlink at ffffffff8b344c8d
#17 [ffff8112247dda98] mds_mfd_close at ffffffff8b384a21
#18 [ffff8112247ddbf8] mds_close at ffffffff8b38c3d0
#19 [ffff8112247ddd08] mds_handle at ffffffff8b35f67b
#20 [ffff8112247dde18] ptlrpc_server_handle_request at ffffffff8b046874
#21 [ffff8112247ddeb8] ptlrpc_main at ffffffff8b047f16
#22 [ffff8112247ddf48] kernel_thread at ffffffff8005dfc1

[Niu Yawei (Inactive)]

Such kind of deadlock only happens when the memory is very tight, I can't think of a good solution for such deadlock so far. Maybe we should just return -ENOMEM and fail the getattr operation but not trigger shrinker in such case?

[Kit Westneat (Inactive)]

Hi Niu,

Would it be possible to return ENOMEM where the lock is held, run the shrinker outside the lock, and then retry the lookup?

Thanks.

[Gerrit Updater]

Niu Yawei ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13743
Subject: LU-3919 lvfs: fid2dentry violates locking order
Project: fs/lustre-release
Branch: b1_8
Current Patch Set: 1
Commit: 1b5c23c2ab0b2c416782e94772ff1f42817a1df3

[Niu Yawei (Inactive)]

Close old 1.8 issue.