[LU-16356] high contention on cdt_request_lock causes clients to hang

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Labels: HSM
• Severity: 3
• Assignee: Sergey Cheremencev
• Reporter: Sergey Cheremencev

Description

If there is above a million entries in cdt_restore_list, it takes a time to go through the all list.

 #4 [ffff8b71d270f900] memcmp at ffffffff8ab7fe1c
 #5 [ffff8b71d270f908] cdt_restore_handle_find at ffffffffc14e7ee9 [mdt]
 #6 [ffff8b71d270f938] mdt_hsm_restore_is_running at ffffffffc14df0c2 [mdt]
 #7 [ffff8b71d270f968] mdt_getattr_internal at ffffffffc14919a1 [mdt]
 #8 [ffff8b71d270f9e0] mdt_getattr_name_lock at ffffffffc1495a7d [mdt]
 #9 [ffff8b71d270fa90] mdt_intent_getattr at ffffffffc149d5d5 [mdt]
#10 [ffff8b71d270fad0] mdt_intent_opc at ffffffffc14926ba [mdt]
#11 [ffff8b71d270fb30] mdt_intent_policy at ffffffffc149a7f4 [mdt]
#12 [ffff8b71d270fb70] ldlm_lock_enqueue at ffffffffc0ff852a [ptlrpc]
#13 [ffff8b71d270fbf0] ldlm_handle_enqueue0 at ffffffffc1020f97 [ptlrpc]
#14 [ffff8b71d270fc80] tgt_enqueue at ffffffffc10ab0f2 [ptlrpc]
#15 [ffff8b71d270fca0] tgt_request_handle at ffffffffc10aff0a [ptlrpc]
#16 [ffff8b71d270fd30] ptlrpc_server_handle_request at ffffffffc1055a56 [ptlrpc]
#17 [ffff8b71d270fde8] ptlrpc_main at ffffffffc1059b35 [ptlrpc]
#18 [ffff8b71d270fec8] kthread at ffffffff8a8c1f81
#19 [ffff8b71d270ff50] ret_from_fork_nospec_begin at ffffffff8af77c1d   

Holding cdt_restore_lock it blocks other tasks that also need cdt_restore_lock:

crash> bt 38174
PID: 38174  TASK: ffff8b722d5fb0c0  CPU: 28  COMMAND: "hsm_cdtr"
 #0 [ffff8b71c31af778] __schedule at ffffffff8af6ab17
 #1 [ffff8b71c31af808] schedule_preempt_disabled at ffffffff8af6bf39
 #2 [ffff8b71c31af818] __mutex_lock_slowpath at ffffffff8af69e87
 #3 [ffff8b71c31af878] mutex_lock at ffffffff8af6926f
 #4 [ffff8b71c31af890] cdt_restore_handle_del at ffffffffc14e8008 [mdt]
 #5 [ffff8b71c31af8c0] mdt_cdt_started_cb at ffffffffc14e8393 [mdt]
 #6 [ffff8b71c31af940] mdt_coordinator_cb at ffffffffc14e8659 [mdt]
 #7 [ffff8b71c31af978] llog_process_thread at ffffffffc0d1a7ff [obdclass]
 #8 [ffff8b71c31afa88] llog_process_or_fork at ffffffffc0d1bae9 [obdclass]
 #9 [ffff8b71c31afaf0] llog_cat_process_cb at ffffffffc0d211ea [obdclass]
#10 [ffff8b71c31afb40] llog_process_thread at ffffffffc0d1a7ff [obdclass]
#11 [ffff8b71c31afc50] llog_process_or_fork at ffffffffc0d1bae9 [obdclass]
#12 [ffff8b71c31afcb8] llog_cat_process_or_fork at ffffffffc0d1d961 [obdclass]
#13 [ffff8b71c31afd30] llog_cat_process at ffffffffc0d1db0e [obdclass]
#14 [ffff8b71c31afd50] cdt_llog_process at ffffffffc14da8be [mdt]
#15 [ffff8b71c31afda0] mdt_coordinator at ffffffffc14e4621 [mdt]
#16 [ffff8b71c31afec8] kthread at ffffffff8a8c1f81
#17 [ffff8b71c31aff50] ret_from_fork_nospec_begin at ffffffff8af77c1d   

Investigating the case from HPe's customer I found there was above a million RESTORE entries in cdt_restore_handle_list:

crash> list -o cdt_restore_handle.crh_list -H 0xffff8b71c5ede158 | wc -l
1218162  

I also tried to analyze the entries but it seems that there are no duplicates in a list, i.e. there are only unique FIDs.

Issue Links

• is related to: LU-16235 cdt_agent_record_hash_add() ASSERTION( carl0->carl_cat_idx == carl1->carl_cat_idx ) failed
• is related to: LU-15132 Parallel data accesses on a release file could hang a MDT

Comments

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54183/
Subject: LU-16356 tests: add trap 0 in cleanup_echo_devs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9b1add59dd64ad3dc794ae852a3c70b7d0a27aad

[Gerrit Updater]

"Sergey Cheremencev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54183
Subject: LU-16356 tests: add trap 0 in cleanup_echo_devs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9e60ef56256009432af11a38bc8ad42fbef197d0

[Alex Zhuravlev]

the patch breaks ost-pools:

=== ost-pools: finish cleanup 07:23:53 (1708845833) ===
lfs setstripe: '0/25' dev not on a mounted Lustre filesystem
lfs setstripe: '/mnt/lustre' is not on a Lustre filesystem: No such device (19)
 ost-pools test_32: @@@@@@ FAIL: error deleting stripe '/mnt/lustre' 
  Trace dump:
  = ./../tests/test-framework.sh:6947:error()
  = ./../tests/test-framework.sh:10806:restore_layout()
  = ./../tests/test-framework.sh:1:exit_status()
  = ost-pools.sh:1977:main()
Dumping lctl log to /tmp/ltest-logs/ost-pools.test_32.*.1708845833.log
Dumping logs only on local client.

bisection is heartless:

COMMIT		TESTED	PASSED	FAILED		COMMIT DESCRIPTION
75dc1bb4bd	5	0	5	BAD	LU-13814 llite: refactor ll_direct_rw_pages
7c98f16b91	5	0	5	BAD	LU-13805 llite: make page_list_{add,del} symmetric
5bc1dd825b	5	0	5	BAD	LU-9839 clio: lov active ios accounting fix
dc13a56187	5	0	5	BAD	LU-16356 hsm: store crh in rhashtable instead of list
5321d51087	5	5	0	GOOD	LU-16096 target: use lsd_reply_data_v1 format by default
d12b18195c	5	5	0	GOOD	LU-17418 libcfs: support debug setup for libcfs modules

the problem is in:

@@ -5048,7 +5048,6 @@ stopall() {
 }
 
 cleanup_echo_devs () {
-       trap 0
        local dev
        local devs=$($LCTL dl | grep echo | awk '{print $4}')

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49284/
Subject: LU-16356 hsm: store crh in rhashtable instead of list
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dc13a56187f6787c23e7ba66c6187a9a6220c5f6

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53735
Subject: LU-16356 hsm: store crh in rhashtable instead of list
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 47c549c029f3bf63bbe2ec420fd8dc633b8c54da

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53661
Subject: LU-16356 hsm: add running ref to the coordinator
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 5361978eab8d95cd9bbf8e592e4d6ec2fb32f9f4

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51256/
Subject: LU-16356 hsm: add running ref to the coordinator
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4512347d6cda68fc8c949db521e00b485258c420