[LU-13413] Lustre soft lockups with peer credit exhaustion

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.12.4
• Labels: ORNL
• Environment: RHEL7.7 running Lustre 2.12 LTS
• Severity: 3
• Assignee: Andreas Dilger
• Reporter: James A Simmons

Description

Currently our production file system is experience peer credit exhaustion which is leading to soft locks. The back trace are attached. Instead of a soft lockups we should be having evicts.

[2642432.333239]  [<ffffffff9477544a>] queued_spin_lock_slowpath+0xb/0xf
[2642432.333241]  [<ffffffff94783330>] _raw_spin_lock+0x20/0x30
[2642432.333258]  [<ffffffffc146202c>] lock_res_and_lock+0x2c/0x50 [ptlrpc]
[2642432.333273]  [<ffffffffc1469c61>] ldlm_lock_enqueue+0x1b1/0xa20 [ptlrpc]
[2642432.333294]  [<ffffffffc14b9891>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
[2642432.333311]  [<ffffffffc1492506>] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc]
[2642432.333332]  [<ffffffffc14bb300>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
[2642432.333357]  [<ffffffffc151acf2>] tgt_enqueue+0x62/0x210 [ptlrpc]
[2642432.333381]  [<ffffffffc1521b0a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[2642432.333404]  [<ffffffffc14fb021>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[2642432.333408]  [<ffffffffc103fbde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[2642432.333428]  [<ffffffffc14c646b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[2642432.333449]  [<ffffffffc14c3285>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[2642432.333451]  [<ffffffff940d3903>] ? __wake_up+0x13/0x20
[2642432.333471]  [<ffffffffc14c9dd4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]

 

Issue Links

• is related to: LU-14221 Client hangs when using DoM with a fixed mdc lru_size
• is related to: LU-12600 Lustre tgt_brw_write() bug