[LU-14951] client hung on ll_file_open/ll_file_release

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Zhenyu Xu

Description

Client hung on mutex taking

1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.475136] INFO: task xxxxxx.x:105352 blocked for more than 120 seconds.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.483154]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.492030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern info kernel [140822.500369] xxxxxx.x  D    0 105352 105340 0x00084000
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.506433] Call Trace:
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.509370]  __schedule+0x2a6/0x700
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.513313]  schedule+0x38/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.516891]  schedule_preempt_disabled+0xa/0x10
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.521843]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.526526]  ? ktime_get+0x36/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.530341]  ll_file_open+0x172/0xbd0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.535177]  ? ll_intent_file_open+0x940/0x940 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.540777]  do_dentry_open+0x132/0x340
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.544987]  finish_open+0x1b/0x30
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.548779]  ll_atomic_open+0x3dd/0x14f0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.553865]  path_openat+0xefd/0x14f0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.557915]  do_filp_open+0x93/0x100
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.561884]  ? __check_object_size+0xa8/0x16b
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.566641]  do_sys_open+0x184/0x220
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.570615]  do_syscall_64+0x5b/0x1a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.574678]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.580139] RIP: 0033:0x7b88f631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.583767] Code: Bad RIP value.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.587388] RSP: 002b:00007fff7ee044f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.595365] RAX: ffffffffffffffda RBX: 00001496d9a8b200 RCX: 000000007b88f631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.602920] RDX: 0000000000000000 RSI: 00001495dcd60b80 RDI: 00000000ffffff9c
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.610477] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.618040] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000077a8d755
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.625608] R13: 0000000077a8d755 R14: 0000000000000001 R15: 00001495dcd60b80
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.633184] INFO: task xxxxxx.x:105354 blocked for more than 120 seconds.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.640922]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.649717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern info kernel [140822.657990] xxxxxx.x  D    0 105354 105340 0x00084000
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.664012] Call Trace:
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.666915]  __schedule+0x2a6/0x700
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.670864]  schedule+0x38/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.674470]  schedule_preempt_disabled+0xa/0x10
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.679472]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.684216]  ? ktime_get+0x36/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.688103]  ll_file_open+0x172/0xbd0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.693026]  ? ll_intent_file_open+0x940/0x940 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.698724]  do_dentry_open+0x132/0x340
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.703035]  finish_open+0x1b/0x30
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.706931]  ll_atomic_open+0x3dd/0x14f0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.712117]  path_openat+0xefd/0x14f0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.716271]  do_filp_open+0x93/0x100
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.720343]  ? __check_object_size+0xa8/0x16b
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.725191]  do_sys_open+0x184/0x220
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.729249]  do_syscall_64+0x5b/0x1a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.733381]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.738899] RIP: 0033:0x7b72f631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.742590] Code: Bad RIP value.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.746277] RSP: 002b:00007ffeeef9c6f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.754312] RAX: ffffffffffffffda RBX: 00001476499c1c00 RCX: 000000007b72f631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.761919] RDX: 0000000000000000 RSI: 00001475458c4f40 RDI: 00000000ffffff9c
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.769531] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.777139] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000041165755
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.784746] R13: 0000000041165755 R14: 0000000000000001 R15: 00001475458c4f40
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.792362] INFO: task xxxxxx.x:105360 blocked for more than 120 seconds.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.800143]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.808973] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern info kernel [140822.817293] xxxxxx.x  D    0 105360 105340 0x00084000
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.823361] Call Trace:
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.826307]  __schedule+0x2a6/0x700
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.830296]  schedule+0x38/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.833932]  schedule_preempt_disabled+0xa/0x10
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.838962]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.843728]  ? ktime_get+0x36/0xa0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.847637]  ll_file_open+0x172/0xbd0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.852589]  ? ll_intent_file_open+0x940/0x940 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.858317]  do_dentry_open+0x132/0x340
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.862655]  finish_open+0x1b/0x30
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.866558]  ll_atomic_open+0x3dd/0x14f0 [lustre]
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.871748]  path_openat+0xefd/0x14f0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.875899]  do_filp_open+0x93/0x100
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.879963]  ? __check_object_size+0xa8/0x16b
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.884809]  do_sys_open+0x184/0x220
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.888868]  do_syscall_64+0x5b/0x1a0
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.893007]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.898524] RIP: 0033:0x7bbea631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.902217] Code: Bad RIP value.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.905905] RSP: 002b:00007ffca6b17770 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.913938] RAX: ffffffffffffffda RBX: 000014bdf3e8a400 RCX: 000000007bbea631
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.921546] RDX: 0000000000000000 RSI: 000014bcf79d7880 RDI: 00000000ffffff9c
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.929158] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.936765] R10: 0000000000000000 R11: 0000000000000293 R12: 00000000418c2755
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern warning kernel [140822.944379] R13: 00000000418c2755 R14: 0000000000000001 R15: 000014bcf79d7880
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.951999] INFO: task xxxxxx.x:105370 blocked for more than 120 seconds.
1628579932 2021 Aug 10 07:18:52 aa5-4072 kern err kernel [140822.959788]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140822.968619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern info kernel [140822.976938] xxxxxx.x  D    0 105370 105340 0x00080000
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140822.983008] Call Trace:
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140822.985958]  __schedule+0x2a6/0x700
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140822.989976]  ? ldlm_lock_decref+0x29/0x70 [ptlrpc]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140822.995272]  schedule+0x38/0xa0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140822.998916]  schedule_preempt_disabled+0xa/0x10
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.003948]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.008742]  ? lprocfs_counter_add+0xd2/0x140 [obdclass]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.014556]  ? ktime_get+0x36/0xa0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.018476]  ll_file_open+0x172/0xbd0 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.023431]  ? ll_intent_file_open+0x940/0x940 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.029155]  do_dentry_open+0x132/0x340
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.033485]  path_openat+0x53e/0x14f0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.037671]  ? __ptlrpc_req_finished+0x6e/0x710 [ptlrpc]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.043497]  ? ll_close_inode_openhandle+0x55d/0xc50 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.049746]  do_filp_open+0x93/0x100
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.053819]  ? __check_object_size+0xa8/0x16b
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.058666]  do_sys_open+0x184/0x220
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.062722]  do_syscall_64+0x5b/0x1a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.066858]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.072380] RIP: 0033:0x7b75a631
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.076078] Code: Bad RIP value.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.079780] RSP: 002b:00007ffcd71bae70 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.087828] RAX: ffffffffffffffda RBX: 0000154efc396e00 RCX: 000000007b75a631
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.095453] RDX: 0000000000000000 RSI: 0000154df65504c0 RDI: 00000000ffffff9c
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.103081] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.110708] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000077486755
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.118337] R13: 0000000077486755 R14: 0000000000000001 R15: 0000154df65504c0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.125960] INFO: task xxxxxx.x:105376 blocked for more than 120 seconds.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.133765]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.142612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern info kernel [140823.150948] xxxxxx.x  D    0 105376 105340 0x00084000
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.157035] Call Trace:
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.159997]  __schedule+0x2a6/0x700
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.164000]  schedule+0x38/0xa0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.167659]  schedule_preempt_disabled+0xa/0x10
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.172702]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.177486]  ? ktime_get+0x36/0xa0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.181415]  ll_file_open+0x172/0xbd0 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.186374]  ? ll_intent_file_open+0x940/0x940 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.192099]  do_dentry_open+0x132/0x340
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.196440]  finish_open+0x1b/0x30
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.200359]  ll_atomic_open+0x3dd/0x14f0 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.205569]  path_openat+0xefd/0x14f0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.209738]  do_filp_open+0x93/0x100
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.213810]  ? __check_object_size+0xa8/0x16b
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.218660]  do_sys_open+0x184/0x220
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.222721]  do_syscall_64+0x5b/0x1a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.226856]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.232379] RIP: 0033:0x7b9da631
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.236080] Code: Bad RIP value.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.239776] RSP: 002b:00007fff7b879af0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.247825] RAX: ffffffffffffffda RBX: 000015385cc83400 RCX: 000000007b9da631
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.255447] RDX: 0000000000000000 RSI: 00001537650fcac0 RDI: 00000000ffffff9c
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.263068] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.270685] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000077766755
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.278300] R13: 0000000077766755 R14: 0000000000000001 R15: 00001537650fcac0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.285921] INFO: task xxxxxx.x:105380 blocked for more than 120 seconds.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.293712]       Tainted: G        W  OE    --------- -t - 4.18.0-240.10.1.el8_3.x86_64 #1
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern err kernel [140823.302555] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern info kernel [140823.310891] xxxxxx.x  D    0 105380 105340 0x00080000
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.316967] Call Trace:
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.319918]  __schedule+0x2a6/0x700
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.323910]  schedule+0x38/0xa0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.327549]  schedule_preempt_disabled+0xa/0x10
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.332575]  __mutex_lock.isra.5+0x2d0/0x4a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.337353]  ll_file_release+0x433/0x8e0 [lustre]
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.342562]  __fput+0xb7/0x230
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.346122]  task_work_run+0x8a/0xb0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.350193]  exit_to_usermode_loop+0xeb/0xf0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.354956]  do_syscall_64+0x198/0x1a0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.359194]  entry_SYSCALL_64_after_hwframe+0x65/0xca
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.364735] RIP: 0033:0x7baea92b
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.368451] Code: Bad RIP value.
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.372167] RSP: 002b:00007ffc5ca25cf8 EFLAGS: 00000206 ORIG_RAX: 0000000000000003
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.380235] RAX: 0000000000000000 RBX: 000015371de8b400 RCX: 000000007baea92b
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.387860] RDX: 000000007bdb1880 RSI: 0000000000000001 RDI: 000000000000003a
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.395480] RBP: 000000007bdb23c0 R08: 00001537309089c0 R09: 000000007ac1cac0
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.403090] R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
1628579933 2021 Aug 10 07:18:53 aa5-4072 kern warning kernel [140823.410695] R13: 0000000077a39940 R14: 0000000077a39998 R15: 0000153623e40e80