|
Maybe a lfsck start/recovery dealdock.
umount is waiting in lfsck_stop() on li_mutex. tgt_recover_2 is in lfsck_start() holding li_mutex. Possibly waiting on lfsck_layout (pid 3013) which is sending notifications but is stuck waiting for import recovery.
Relevant backtraces:
|
|
Fan Yong, does this look familiar?
|
|
According to current implementation (master), the LFSCK notify RPC is interruptible, so would you please to try "kill -9 3013" to check whether the layout LFSCK assistant engine can be interrupted? If yes, then the lfsck_start can go ahead (exit with failure), and then the lfsck_stop will not be blocked.
|
|
Okay, so you want lfsck_layout process killed if this happens again?
|
|
Yes, please kill the blocked (on recovery) LFSCK assistant thread, such as lfsck_layout or lfsck_namespace. It is expected to work, otherwise, we need to find out the solution.
|
|
It does not work.
[root@soak-8 ~]# ps -ef |grep lfsck
root 6695 6693 0 04:35 ? 00:00:00 bash -c ??lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A ??if [[ $? != 0 ]]; then ???lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace,layout -A ???if [[ $? != 0 ]]; then ????lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace -A ???fi ??fi
root 6713 6695 0 04:35 ? 00:00:00 lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A
root 6714 2 0 04:35 ? 00:00:00 [lfsck]
root 6716 2 0 04:35 ? 00:00:00 [lfsck_layout]
root 53709 53676 0 16:09 pts/0 00:00:00 lctl lfsck_stop -M soaked-MDT0000
root 53748 53717 0 16:10 pts/1 00:00:00 grep --color=auto lfsck
[root@soak-8 ~]# kill -9 6716
[root@soak-8 ~]# kill -9 6716
[root@soak-8 ~]# ps -ef |grep lfsck
root 6695 6693 0 04:35 ? 00:00:00 bash -c ??lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A ??if [[ $? != 0 ]]; then ???lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace,layout -A ???if [[ $? != 0 ]]; then ????lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace -A ???fi ??fi
root 6713 6695 0 04:35 ? 00:00:00 lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A
root 6714 2 0 04:35 ? 00:00:00 [lfsck]
root 6716 2 0 04:35 ? 00:00:00 [lfsck_layout]
root 53709 53676 0 16:09 pts/0 00:00:00 lctl lfsck_stop -M soaked-MDT0000
root 53754 53717 0 16:11 pts/1 00:00:00 grep --color=auto lfsck
Process is un-killable, soak-8 is wedged. I can leave it like this if you want to login.
|
|
Just login soak-8, but seems reboot.
This situation is more complex than LU-6684. According to current (master) implementation, when lfsck_stop, it will send SIGINT signal to the LFSCK engines, then even if the LFSCK engines are blocked by some RPCs, it still can be waken up and exit. But in this case, the lfsck_stop was blocked on the mutex, then it has no chance to send the SIGINT single.
|
|
Sorry, I'll let you know when the next hang happens. I should have dumped stacks before the reboot.
|
|
I think i have re-produced the hang, I'll leave it like this until you have a look.
Sigh, it finally exited. Trying again.
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/30420
Subject: LU-10321 lfsck: allow to stop the in-starting lfsck
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7817928a62c8a212f5c2b040587042c193bd5c1a
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/30422
Subject: LU-10321 lfsck: allow to stop the in-starting lfsck
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 162e2f5c04b32904053ac31795450eb9da521c00
|
|
I think there is already an option no scrub to prevent Oi scrub from starting.
However, it would also be useful if it was possible to force LFSCK to abort the running scan at startup, in case there is a problem like this at a customer. Is “no scrub” along with “lctl lfsck_stop —reset” enough to kill the running LFSCK, and erase the saved state?
|
|
For this case, it is NOT the LFSCK auto-resumed from the last check-point, instead, it was started manually (or by scripts) after MDT failover. So no related with mount option.
|
|
Hit the problem again last night. System completed a failback, MDTs are all mounted in the proper place, lfsck is wedged. Will leave the system if you wish to have a look.
|
|
Dumped lustre logs from all MDS, output in /scratch/results/soak/soak-X-hung.lfsck.Dec08.txt
Dumped stacks on all MDS, output in console logs. Crash-dumped all MDS, results on Spirit.
Restarting.
|
|
Had a simular hang on soak-3 and soak-6 (OSS) during umount. Dumped lustre-logs, attached. Also dumped stack traces and crash dumped both nodes, output available on spirit.
|
|
cliffw, as you said on Skype, there are multiple issues during current Spirit tests. The original hung happened on the MDT because of the blocked uninterrupted LFSCK. The patch 30420 is used for resolving such trouble. But it does not means the Spirit will not hung after applying the patch, because some other issues may also block the system. For the new hung, it happened on the OST side, different from the original LFSCK hung. So would you please to check whether the original LFSCK hung issue resolved or not? Thanks!
|
|
We have not seen lfsck hangs with MDT failover after the patch. The two hangs occurred with OST failover. After the most recent hang, i was able to reboot/remount the system and then I could start and stop lfsck without the hang.
|
|
According to the logs, three were new LFSCK started just after lfsck_stop during the MDT umount. Then nobody will stop the new triggered LFSCK as to the MDT cannot umount. I will make patch to resolve the race condition.
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/30513
Subject: LU-10321 lfsck: not start lfsck during umount
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6263064b369dd4fddbb0dfa9ab49013a0d791629
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/30514
Subject: LU-10321 lfsck: not start lfsck during umount
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 2022d417ddaf663dc7addb5389acade0390996e5
|
|
We just hit this again on master, so we need a version of the patch for master.
|
|
Okay, we have an IB build so we will test this.
|
|
We are still hitting this issue, with the patch. Soak-10 just hit it, crash dump vmcore-dmesg and lustre-log attached
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/30420/
Subject: LU-10321 lfsck: allow to stop the in-starting lfsck
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9c9a05fee6c0fce557dfa578ff7116b905d4e00a
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/30513/
Subject: LU-10321 lfsck: not start lfsck during umount
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0cabe06be1fcc6c3ff690889b1e86bbe4690b854
|
|
Patches have been landed to master, other LFSCK start trouble will be handled via LU-10419
|
|
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/30422/
Subject: LU-10321 lfsck: allow to stop the in-starting lfsck
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 147b2b4177f9f8d3c4407cd74bb1b97a5c5d7f53
|
|
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/30514/
Subject: LU-10321 lfsck: not start lfsck during umount
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: e82d8ad284fd37bbc4863723aa42c927d456fae6
|
Generated at Sat Feb 10 02:34:00 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
soak-10.umount.hang.txt.gz
soak-11.umount.hang.txt