-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Stuck umount stack looks like :
PID: 3227430 TASK: ff452bb41509af80 CPU: 30 COMMAND: "umount"
#0 [ff67e31e264ffcc0] __schedule at ffffffffa365af2c
#1 [ff67e31e264ffd68] schedule at ffffffffa365b3a3
#2 [ff67e31e264ffd88] schedule_timeout at ffffffffa3662425
#3 [ff67e31e264ffde0] schedule_timeout_uninterruptible at ffffffffa36625ae
#4 [ff67e31e264ffdf0] ll_kill_super at ffffffffc227ebfd [lustre]
#5 [ff67e31e264ffe10] lustre_kill_super at ffffffffc22b52b6 [lustre]
#6 [ff67e31e264ffe28] deactivate_locked_super at ffffffffa28f8ee2
#7 [ff67e31e264ffe48] deactivate_super at ffffffffa28f8fc6
#8 [ff67e31e264ffe60] cleanup_mnt at ffffffffa2929733
#9 [ff67e31e264ffe90] __cleanup_mnt at ffffffffa2929862
#10 [ff67e31e264ffea0] task_work_run at ffffffffa253a09e
#11 [ff67e31e264ffec8] syscall_exit_to_user_mode at ffffffffa3653198
#12 [ff67e31e264ffef0] do_syscall_64 at ffffffffa364c84c
#13 [ff67e31e264fff50] entry_SYSCALL_64_after_hwframe at ffffffffa3800130
RIP: 00007e4b4d72aa2b RSP: 00007ffd843a1b78 RFLAGS: 00000246
RAX: 0000000000000000 RBX: 00005f7166a95a00 RCX: 00007e4b4d72aa2b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005f7166a9f320
RBP: 00007ffd843a1c50 R8: 00007e4b4d803b20 R9: 0000000000000020
R10: 0000000000000001 R11: 0000000000000246 R12: 00005f7166a95b00
R13: 0000000000000000 R14: 00005f7166a9f320 R15: 00005f7166a9f610
ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b
Orphan statahead threads stacks looks like :
PID: 354918 TASK: ff452bffd67a2f80 CPU: 10 COMMAND: "ll_sa_354917" #0 [ff67e31e24d9fc88] __schedule at ffffffffa365af2c #1 [ff67e31e24d9fd30] schedule at ffffffffa365b3a3 #2 [ff67e31e24d9fd50] schedule_timeout at ffffffffa3662425 #3 [ff67e31e24d9fda8] ll_statahead_thread at ffffffffc22b8c2b [lustre] #4 [ff67e31e24d9fee8] kthread at ffffffffa253e2cf #5 [ff67e31e24d9ff28] ret_from_fork at ffffffffa246a6f4 #6 [ff67e31e24d9ff50] ret_from_fork_asm at ffffffffa24058eb PID: 554465 TASK: ff452bb2064b2f80 CPU: 92 COMMAND: "ll_sa_551592" #0 [ff67e31e27767c88] __schedule at ffffffffa365af2c #1 [ff67e31e27767d30] schedule at ffffffffa365b3a3 #2 [ff67e31e27767d50] schedule_timeout at ffffffffa3662425 #3 [ff67e31e27767da8] ll_statahead_thread at ffffffffc22b8c2b [lustre] #4 [ff67e31e27767ee8] kthread at ffffffffa253e2cf #5 [ff67e31e27767f28] ret_from_fork at ffffffffa246a6f4 #6 [ff67e31e27767f50] ret_from_fork_asm at ffffffffa24058eb PID: 648258 TASK: ff452bef5f42af80 CPU: 100 COMMAND: "ll_sa_648237" #0 [ff67e31e21dd7c88] __schedule at ffffffffa365af2c #1 [ff67e31e21dd7d30] schedule at ffffffffa365b3a3 #2 [ff67e31e21dd7d50] schedule_timeout at ffffffffa3662425 #3 [ff67e31e21dd7da8] ll_statahead_thread at ffffffffc22b8c2b [lustre] #4 [ff67e31e21dd7ee8] kthread at ffffffffa253e2cf #5 [ff67e31e21dd7f28] ret_from_fork at ffffffffa246a6f4 #6 [ff67e31e21dd7f50] ret_from_fork_asm at ffffffffa24058eb PID: 1797425 TASK: ff452bccf825af80 CPU: 38 COMMAND: "ll_sa_1797415" #0 [ff67e31e25367c88] __schedule at ffffffffa365af2c #1 [ff67e31e25367d30] schedule at ffffffffa365b3a3 #2 [ff67e31e25367d50] schedule_timeout at ffffffffa3662425 #3 [ff67e31e25367da8] ll_statahead_thread at ffffffffc22b8c2b [lustre] #4 [ff67e31e25367ee8] kthread at ffffffffa253e2cf #5 [ff67e31e25367f28] ret_from_fork at ffffffffa246a6f4 #6 [ff67e31e25367f50] ret_from_fork_asm at ffffffffa24058eb PID: 2763919 TASK: ff452bc318f18000 CPU: 40 COMMAND: "ll_sa_2763917" #0 [ff67e31e33f4fc88] __schedule at ffffffffa365af2c #1 [ff67e31e33f4fd30] schedule at ffffffffa365b3a3 #2 [ff67e31e33f4fd50] schedule_timeout at ffffffffa3662425 #3 [ff67e31e33f4fda8] ll_statahead_thread at ffffffffc22b8c2b [lustre] #4 [ff67e31e33f4fee8] kthread at ffffffffa253e2cf #5 [ff67e31e33f4ff28] ret_from_fork at ffffffffa246a6f4 #6 [ff67e31e33f4ff50] ret_from_fork_asm at ffffffffa24058eb PID: 354919 TASK: ff452bcde8d42f80 CPU: 28 COMMAND: "ll_agl_354917" #0 [ff67e31e2039bde8] __schedule at ffffffffa365af2c #1 [ff67e31e2039be90] schedule at ffffffffa365b3a3 #2 [ff67e31e2039beb0] ll_agl_thread at ffffffffc22b6aa9 [lustre] #3 [ff67e31e2039bee8] kthread at ffffffffa253e2cf #4 [ff67e31e2039bf28] ret_from_fork at ffffffffa246a6f4 #5 [ff67e31e2039bf50] ret_from_fork_asm at ffffffffa24058eb PID: 554466 TASK: ff452bf238c28000 CPU: 73 COMMAND: "ll_agl_551592" #0 [ff67e31e2776fde8] __schedule at ffffffffa365af2c #1 [ff67e31e2776fe90] schedule at ffffffffa365b3a3 #2 [ff67e31e2776feb0] ll_agl_thread at ffffffffc22b6aa9 [lustre] #3 [ff67e31e2776fee8] kthread at ffffffffa253e2cf #4 [ff67e31e2776ff28] ret_from_fork at ffffffffa246a6f4 #5 [ff67e31e2776ff50] ret_from_fork_asm at ffffffffa24058eb PID: 648259 TASK: ff452bb23fd90000 CPU: 58 COMMAND: "ll_agl_648237" #0 [ff67e31e21ddfde8] __schedule at ffffffffa365af2c #1 [ff67e31e21ddfe90] schedule at ffffffffa365b3a3 #2 [ff67e31e21ddfeb0] ll_agl_thread at ffffffffc22b6aa9 [lustre] #3 [ff67e31e21ddfee8] kthread at ffffffffa253e2cf #4 [ff67e31e21ddff28] ret_from_fork at ffffffffa246a6f4 #5 [ff67e31e21ddff50] ret_from_fork_asm at ffffffffa24058eb PID: 1797426 TASK: ff452be7c8c22f80 CPU: 62 COMMAND: "ll_agl_1797415" #0 [ff67e31e25a07de8] __schedule at ffffffffa365af2c #1 [ff67e31e25a07e90] schedule at ffffffffa365b3a3 #2 [ff67e31e25a07eb0] ll_agl_thread at ffffffffc22b6aa9 [lustre] #3 [ff67e31e25a07ee8] kthread at ffffffffa253e2cf #4 [ff67e31e25a07f28] ret_from_fork at ffffffffa246a6f4 #5 [ff67e31e25a07f50] ret_from_fork_asm at ffffffffa24058eb PID: 2763920 TASK: ff452c09beac8000 CPU: 38 COMMAND: "ll_agl_2763917" #0 [ff67e31e32913de8] __schedule at ffffffffa365af2c #1 [ff67e31e32913e90] schedule at ffffffffa365b3a3 #2 [ff67e31e32913eb0] ll_agl_thread at ffffffffc22b6aa9 [lustre] #3 [ff67e31e32913ee8] kthread at ffffffffa253e2cf #4 [ff67e31e32913f28] ret_from_fork at ffffffffa246a6f4 #5 [ff67e31e32913f50] ret_from_fork_asm at ffffffffa24058eb
Each of the "parent" process, with <PID> from ll_sa<PID{_}, is no longer present.
Disassembly of ll_statahead_thread() code shows that all ll_sa<PID>_ thread are looping around shedule_timeout() in function epilog part waiting for the close()/release of the file to occur :
................
/* statahead is finished, but statahead entries need to be cached, wait for file release to stop me. */ while (( {set_current_state(TASK_IDLE); /* matches smp_store_release() in ll_deauthorize_statahead() */ smp_load_acquire(&sai->sai_task); } )) Unknown macro: { if (sa_has_callback(sai)) { __set_current_state(TASK_RUNNING); sa_handle_callback(sai); } else { /* Check regularly in case we have missed the wakeup */ schedule_timeout(msecs_to_jiffies(100)); } } .................
"lustre/llite/statahead.c" 1823 lines {}66%{} 1204,0-1 68%
I have also examined the directory inodes being handled by each of the ll_sa<PID>_ and they all have their lli_opendir_key/lli_opendir_pid/lli_sa_enabled fields set to 0 and they all point point to a struct ll_statahead_info where sai_task field/pointer is populated and points to one of the present ll_sa<PID>_ task struct.
So, all of these points tend to indicate that file has been close()ed/release (and ll_deauthorize_statahead() is called) during the timing window where ll_sa<PID>_ Kthread has been created and sai_task is being initialized unprotected in start_statahead_thread().