[LU-10419] LFSCK fails to start, hangs systems.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.5
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.2, Lustre 2.10.3
• Labels: soak
• Environment: Soak performance cluster - Lustre version=2.10.2_4_gb151f34
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Cliff White (Inactive)

Description

We do OSS failover, trigger LFSCK:

lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A{code]

The lfsck start hangs, lfsck is not started, the clients wedge in state 'comp' the entire system wedges. I have dumped Lustre Logs from all MDS, attached. I have crash-dumped all the MDT nodes and the dumps are available on Spirit. lfsck_layout is unkillable.

Issue Links

• is duplicated by: LU-10036 lfsck stuck in init on multiple MDS

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31600/
Subject: Revert "LU-10419 lfsck: skip dead target"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9ba637b8949b1b8a5f2506e654a9b62d5c0cc245

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31600
Subject: Revert "LU-10419 lfsck: skip dead target"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1387fa1c012dfdf5eb4f90efeb06edd45788064f

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31475/
Subject: LU-10419 lfsck: skip dead target
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 012834c5e7c7be50ff117cee4ac473d7fee4294d

[Cliff White (Inactive)]

With the current patch, lfsck does not stop. Currently also having mount timeouts. I have crashed dumped soak-8 while lfsck was hanging, logs are available on spirit.
/scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2018-03-06-19:16:47

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31475
Subject: LU-10419 lfsck: skip dead target
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: be9f2eedf5039fa6308460aca6a84daa6b8003b1

[Cliff White (Inactive)]

Logs are on spirit /scratch/logs/syslogs and /scratch/logs/console. The crash dumps are in /scratch/dumps on spirit.

[nasf (Inactive)]

cliffw,

Where can I get related logs?

Thanks!

[Cliff White (Inactive)]

Seeing this again on DNE-enable system. version=2.10.57_58_gf24340c
I can crash dump systems if desired

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30768/
Subject: LU-10419 lfsck: no delay for notify RPC
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 39816213632cf9083530f1a8b644459d13e3c980