[LU-10419] LFSCK fails to start, hangs systems.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.5
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.2, Lustre 2.10.3
• Labels: soak
• Environment: Soak performance cluster - Lustre version=2.10.2_4_gb151f34
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Cliff White (Inactive)

Description

We do OSS failover, trigger LFSCK:

lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A{code]

The lfsck start hangs, lfsck is not started, the clients wedge in state 'comp' the entire system wedges. I have dumped Lustre Logs from all MDS, attached. I have crash-dumped all the MDT nodes and the dumps are available on Spirit. lfsck_layout is unkillable.

Issue Links

• is duplicated by: LU-10036 lfsck stuck in init on multiple MDS

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/30831/
Subject: LU-10419 lfsck: no delay for notify RPC
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 9fef9ad10b26a4338c22105e66308ead5408173e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31627/
Subject: LU-10419 lfsck: signal master engine when stop
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1ece380412efd5dba2a8c345830f0456a4922301

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31627
Subject: LU-10419 lfsck: single master engine when stop
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e3e7d1a41711cfb0a12b941a88bf8c0bf3b4cc89

[nasf (Inactive)]

With the current patch, lfsck does not stop. Currently also having mount timeouts. I have crashed dumped soak-8 while lfsck was hanging, logs are available on spirit.
/scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2018-03-06-19:16:47

The LFSCK master engine was blocked when sending OUT_ATTR_GET RPC to MDT2 that may be offline or in recovery. We expect the lfsck_stop() can wakeup the blocked LFSCK engines and make them to exit, but we only single (SIGINT) the LFSCK assistant engines, forget to do that for the LFSCK master engine.

 

So the trouble is not related with the patch https://review.whamcloud.com/31475/.

I will make another patch to notify the master engine when lfsck_stop().

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31600/
Subject: Revert "LU-10419 lfsck: skip dead target"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9ba637b8949b1b8a5f2506e654a9b62d5c0cc245

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31600
Subject: Revert "LU-10419 lfsck: skip dead target"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1387fa1c012dfdf5eb4f90efeb06edd45788064f

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31475/
Subject: LU-10419 lfsck: skip dead target
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 012834c5e7c7be50ff117cee4ac473d7fee4294d

[Cliff White (Inactive)]

With the current patch, lfsck does not stop. Currently also having mount timeouts. I have crashed dumped soak-8 while lfsck was hanging, logs are available on spirit.
/scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2018-03-06-19:16:47

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31475
Subject: LU-10419 lfsck: skip dead target
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: be9f2eedf5039fa6308460aca6a84daa6b8003b1