[LU-7696] sanity-lfsck test 18d fails with "(3) MDS1 is not the expected 'completed' "

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0, Lustre 2.10.0, Lustre 2.11.0, Lustre 2.12.1
• Labels: lfsck
• Environment: autotest review-dne-part-2
• Severity: 3
• Assignee: WC Triage
• Reporter: James Nunez (Inactive)

Description

sanity-lfsck test 18d has failed a few times during review-dne-part-2 with the error

'(3) MDS4 is not the expected 'completed'' 

From the client test log we can see that LFSCK is still scanning the file system:

CMD: shadow-21vm8 /usr/sbin/lctl get_param -n 			mdd.lustre-MDT0003.lfsck_layout |
			awk '/^status/ { print \$2 }'
CMD: shadow-21vm8 /usr/sbin/lctl get_param -n 			mdd.lustre-MDT0003.lfsck_layout |
			awk '/^status/ { print \$2 }'
Update not seen after 120s: wanted 'completed' got 'scanning-phase2'
 sanity-lfsck test_18d: @@@@@@ FAIL: (3) MDS4 is not the expected 'completed' 

Logs for recent failures with this error are at:
2015-12-12 07:18:52 - https://testing.hpdd.intel.com/test_sets/7bbc6232-a0e2-11e5-9d88-5254006e85c2
2015-12-22 14:14:50 - https://testing.hpdd.intel.com/test_sets/0bb9a35c-a8c9-11e5-b5b1-5254006e85c2
2016-01-22 03:29:28 - https://testing.hpdd.intel.com/test_sets/a2b59d3e-c0fb-11e5-8d88-5254006e85c2

Note: this failure is different from LU-5487.

Comments

[Sarah Liu]

on 2_12 branch
https://testing.whamcloud.com/test_sets/4e94da42-5b23-11e9-a256-52540065bddc

[Jian Yu]

One more failure instance on master branch:
https://testing.hpdd.intel.com/test_sets/36eb8f70-233f-11e8-8d2f-52540065bddc

[Jinshan Xiong (Inactive)]

new occurrence: https://testing.hpdd.intel.com/test_sessions/27106aa3-c69f-477f-9e2a-8c8b91d1fe73

[Jian Yu]

More failure instances on master branch (review-dne-part-2 test session):
https://testing.hpdd.intel.com/test_sets/518b5194-9d61-11e7-b778-5254006e85c2
https://testing.hpdd.intel.com/test_sets/cdb8b8ec-98cf-11e7-b778-5254006e85c2

[Jian Yu]

The similar failure occurred on master branch:
https://testing.hpdd.intel.com/test_sets/c31b4cf2-a1b7-11e6-9648-5254006e85c2

[Richard Henwood (Inactive)]

Hit this today (feb 2nd 2016) with the canary patch.

https://testing.hpdd.intel.com/test_sessions/e7ad9502-c913-11e5-aaa9-5254006e85c2