[LU-4801] spin lock contention in lock_res_and_lock

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.4.1
• Labels: llnl
• Environment: lustre-2.4.0-26chaos
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Ned Bass (Inactive)

Description

Our MDS experienced severe lock contention in lock_res_and_lock(). This had a large impact on client responsiveness because service threads were starved for CPU time. We have not yet identified the client workload that caused this problem. All active tasks had stack traces like this, but would eventually get scheduled out.

 ...
__spin_lock
lock_res_and_lock
ldlm_handle_enqueue0
mdt_handle_common
mds_regular_handle
ptlrpc_server_handle_request
...

This raises the question of why the ldlm resource lock needs to be a spinlock. Couldn't we avoid this issue by converting it to a mutex? This question was raised in LU-3504.

Issue Links

• is related to: LU-12542 LDLM improvements form linux lustre client work
• is related to: LU-3504 MDS: All cores spinning on ldlm lock in lock_res_and_lock

Comments

[James A Simmons]

Patch has landed. We will look to deploy this on our production systems soon. Any further work needed we can reopen this ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39811/
Subject: LU-4801 ldlm: discard l_lock from struct ldlm_lock.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bb6edb7b8eeec65f46f8eaeb135e5dde13bf7ad8

[Gerrit Updater]

Neil Brown ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39811
Subject: LU-4801 ldlm: discard l_lock from struct ldlm_lock.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c634dbeb9e576574a903f83dfb9cf39cf8d5468b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38238/
Subject: Revert "LU-4801 ldlm: discard l_lock from struct ldlm_lock."
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9051844cec34ab4f3427adc28bc1948706f5ffc2

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38238
Subject: Revert "LU-4801 ldlm: discard l_lock from struct ldlm_lock."
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 32095b5717954fa7260c9d6e369a208395bc39da

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35483/
Subject: LU-4801 ldlm: discard l_lock from struct ldlm_lock.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0584eb73dbb5b4c710a8c7eb1553ed5dad0c18d8

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35484
Subject: LU-4801 ldlm: don't access l_resource when not locked.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5ea026bb3eb9020233569659189850519bc99a17

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35483
Subject: LU-4801 ldlm: discard l_lock from struct ldlm_lock.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c7e16e62306abdb62f1039957573400c9114ea3f

[James A Simmons]

Neil pushed some patches to address. I will push

[Ned Bass (Inactive)]

Oleg, it may not be right away, but we'll get this scheduled for testing.