[LU-7428] conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0
• Labels: p4hc
• Severity: 3
• Assignee: James A Simmons
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/5d42a610-8187-11e5-a41e-5254006e85c2.

The sub-test test_84 failed with the following error:

CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Update not seen after 90s: wanted '' got 'lustre:MDT0000'
 conf-sanity test_84: @@@@@@ FAIL: /dev/lvm-Role_MDS/P1 failed to initialize! 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4843:error()
  = /usr/lib64/lustre/tests/test-framework.sh:1270:mount_facet()
  = /usr/lib64/lustre/tests/test-framework.sh:1188:mount_facets()
  = /usr/lib64/lustre/tests/test-framework.sh:2513:facet_failover()
  = /usr/lib64/lustre/tests/conf-sanity.sh:5594:test_84()
  = /usr/lib64/lustre/tests/test-framework.sh:5090:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5127:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4992:run_test()

Please provide additional information about the failure here.

Info required for matching: conf-sanity 84
Info required for matching: replay-dual 0a

Issue Links

• is duplicated by: LU-7481 Failover: recovery-mds-scale test_failover_mds: /dev/lvm-Role_MDS/P1 failed to initialize!
• is related to: LU-6992 recovery-random-scale test_fail_client_mds: test_fail_client_mds returned 254
• is related to: LU-7222 conf-sanity test_84: invalid llog tail at log id 0x4:10/0 offset 16384
• is related to: LU-7368 e2fsck unsafe to interrupt with quota enabled
• is related to: LU-6789 Interop 2.5.3<->master conf-sanity test_84: completed_clients != 1/2: 2/2
• is related to: LU-7361 conf-sanity test_84: Error: 'recovery_duration > recovery_time_hard + wrap up'
• is related to: LU-7492 conf-sanity_87 test failed conf-sanity test_87: @@@@@@ FAIL: on-disk parameter not replaced via tunefs
• is related to: LU-7100 conf-sanity test_84 LBUGS with “(llog_osd.c:811:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index )”
• is related to: LU-7509 Failover: recovery-random-scale test_fail_client_mds: /dev/lvm-Role_MDS/P1 failed to initialize!
• is related to: LU-7364 conf-sanity test_84 fails with llog_process_thread()) invalid length 0 in llog record for index 0/68 'Restart of mds1 failed!'
• is related to: LU-6029 conf-sanity test_84: recovery_duration > recovery_time_hard
• is related to: LU-7169 conf-sanity 84 restart mds1 failed
• is related to: LU-7097 conf-sanity test_84 (check recovery_time_hard) fails on DNE setup

Comments

[Andreas Dilger]

Reopen until patch enabling test_84 actually lands.

[Hongchao Zhang]

the patch http://review.whamcloud.com/#/c/20194/ to remove test 84 has been refreshed, and have passed the tests in Maloo now.

[John Hammond]

Did the landing of http://review.whamcloud.com/20586/ resolve this issue? I see that we still have 84 in ALWAYS_EXCEPT.

[Peter Jones]

Landed for 2.9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20586/
Subject: LU-7428 osd: set device read-only correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a079ade7913b923b795ea5c01df4e69bf1a87691

[Andreas Dilger]

No, this won't replace LU-684.

This patch is to (hopefully) fix a problem where the device is sync'd and set read-only, but loses some recent writes, for an unknown reason. This shows up with a variety of different symptoms, and may be a result of bad interactions with LVM and VM virtual block devices, or it may be caused by the dev readonly patches.

[James A Simmons]

Does this patch mean we don't need LU-684 anymore?

[Hongchao Zhang]

the patch ported from MRP-2135 (https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b)
is tracked at http://review.whamcloud.com/20586

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20586
Subject: LU-7428 osd: set rdonly correctly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 521b8290fbf0b47d4ad03272a206d038f648db2d

[Gerrit Updater]

Gu Zheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20535
Subject: LU-7428 osd: freeze fs before set device readonly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cb2a762f99cef9ff86dc76445248f91b40f0199b

[Andreas Dilger]

It might be worthwhile to test the patch from https://github.com/Xyratex/lustre-stable/commit/6197a27f174e683d3c66137db8976bddc7ef179b to see if that is fixing the problem? I think that patch could be simplified to just call sb->s_op->s_freeze() before marking the device read-only.