[LU-6684] lctl lfsck_stop hangs

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Frederik Ferner (Inactive)

Description

As mentioned in LU-6683, I ran into a situation where lctl lfsck_stop just hangs indefinitely.

I have managed to reproduce this twice:

start lfsck (using lctl lfsck_start -M play01-MDT0000 -t layout), this crashes the OSS servers, reboot the servers and restart the OSTs. Attempting to stop the lfsck in this state just hangs. I have waited >1h and it was still hanging. Unmounting the MDT in this situation also appears to be hanging (after 30 minutes I power cycled the MDS).

Issue Links

• is duplicated by: LU-7662 lfsck don't complete
• is related to: LU-10321 MDS - umount hangs during failback

Comments

[Saurabh Tandan (Inactive)]

Another instance found for tag 2.7.66 for Full - EL6.7 Server/EL6.7 Client
On master, build# 3314
https://testing.hpdd.intel.com/test_sets/35490a0c-ca6e-11e5-9215-5254006e85c2
Date : 02/02/2016 Time: 9:20 am MST

[nasf (Inactive)]

The patch has been landed to master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/18082/
Subject: LU-6684 lfsck: set the lfsck notify as interruptable
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 069a9cf551c2e985ea254a1c570b22ed1d72d914

[Jian Yu]

This is blocking patch review testing on master branch:
https://testing.hpdd.intel.com/test_sets/a29caebe-c709-11e5-9b6d-5254006e85c2
https://testing.hpdd.intel.com/test_sets/fbfee2be-c70f-11e5-a037-5254006e85c2

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/150c07e2-c575-11e5-825e-5254006e85c2

[James A Simmons]

This is also delaying the landing of several patches.

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/85d45ece-c0bc-11e5-9620-5254006e85c2

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18082
Subject: LU-6684 lfsck: set the lfsck notify as interruptable
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 68c078328be253735658fcf43fa98afff936ec6c

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18059
Subject: Revert "LU-6684 lfsck: stop lfsck even if some servers offline"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2505fd07b29ebfddcd29f16954908f6fe4670276

[James Nunez (Inactive)]

More failures on master and all have the previous patch landed for this ticket:
2016-01-15 15:29:21 - https://testing.hpdd.intel.com/test_sets/48126330-bbce-11e5-8506-5254006e85c2
2016-01-15 20:20:20 - https://testing.hpdd.intel.com/test_sets/7ec04c5e-bbfa-11e5-acbb-5254006e85c2
2016-01-16 00:40:11 - https://testing.hpdd.intel.com/test_sets/4988556c-bc05-11e5-8f65-5254006e85c2
2016-01-18 22:08:02 - https://testing.hpdd.intel.com/test_sets/3a54dfd8-be63-11e5-92e8-5254006e85c2
2016-01-18 22:59:29 - https://testing.hpdd.intel.com/test_sets/642d055a-be69-11e5-92e8-5254006e85c2
2016-01-18 23:21:01 - https://testing.hpdd.intel.com/test_sets/c75e157e-be6e-11e5-b113-5254006e85c2
2016-01-19 07:37:19 - https://testing.hpdd.intel.com/test_sets/325db7ae-beb4-11e5-8c8a-5254006e85c2
2016-01-19 12:10:06 - https://testing.hpdd.intel.com/test_sets/144d9d36-bed9-11e5-ad7e-5254006e85c2
2016-01-19 22:11:45 - https://testing.hpdd.intel.com/test_sets/a2f0fede-bf2e-11e5-a659-5254006e85c2
2016-01-19 22:26:33 - https://testing.hpdd.intel.com/test_sets/dc0ed974-bf2f-11e5-8f04-5254006e85c2
2016-01-19 23:59:17 - https://testing.hpdd.intel.com/test_sets/01d6b960-bf3f-11e5-8f04-5254006e85c2
2016-01-21 11:12:25 - https://testing.hpdd.intel.com/test_sets/cd343b46-c061-11e5-a8e5-5254006e85c2
2016-01-21 13:03:12 - https://testing.hpdd.intel.com/test_sets/c0b04f0e-c070-11e5-956d-5254006e85c2
2016-01-21 14:41:59 - https://testing.hpdd.intel.com/test_sets/e4cffce0-c07f-11e5-a8e5-5254006e85c2
2016-01-21 21:40:44 - https://testing.hpdd.intel.com/test_sets/85d45ece-c0bc-11e5-9620-5254006e85c2
2016-01-22 03:45:40 - https://testing.hpdd.intel.com/test_sets/abf22bb0-c0ec-11e5-8d88-5254006e85c2

[Andreas Dilger]

And I verified that these two failures are on commits that include the fix that was recently landed here.