[LU-7428] conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0
• Labels: p4hc
• Severity: 3
• Assignee: James A Simmons
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/5d42a610-8187-11e5-a41e-5254006e85c2.

The sub-test test_84 failed with the following error:

CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: shadow-10vm4 e2label /dev/lvm-Role_MDS/P1 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Update not seen after 90s: wanted '' got 'lustre:MDT0000'
 conf-sanity test_84: @@@@@@ FAIL: /dev/lvm-Role_MDS/P1 failed to initialize! 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4843:error()
  = /usr/lib64/lustre/tests/test-framework.sh:1270:mount_facet()
  = /usr/lib64/lustre/tests/test-framework.sh:1188:mount_facets()
  = /usr/lib64/lustre/tests/test-framework.sh:2513:facet_failover()
  = /usr/lib64/lustre/tests/conf-sanity.sh:5594:test_84()
  = /usr/lib64/lustre/tests/test-framework.sh:5090:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5127:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4992:run_test()

Please provide additional information about the failure here.

Info required for matching: conf-sanity 84
Info required for matching: replay-dual 0a

Issue Links

• is duplicated by: LU-7481 Failover: recovery-mds-scale test_failover_mds: /dev/lvm-Role_MDS/P1 failed to initialize!
• is related to: LU-6992 recovery-random-scale test_fail_client_mds: test_fail_client_mds returned 254
• is related to: LU-7222 conf-sanity test_84: invalid llog tail at log id 0x4:10/0 offset 16384
• is related to: LU-7368 e2fsck unsafe to interrupt with quota enabled
• is related to: LU-6789 Interop 2.5.3<->master conf-sanity test_84: completed_clients != 1/2: 2/2
• is related to: LU-7361 conf-sanity test_84: Error: 'recovery_duration > recovery_time_hard + wrap up'
• is related to: LU-7492 conf-sanity_87 test failed conf-sanity test_87: @@@@@@ FAIL: on-disk parameter not replaced via tunefs
• is related to: LU-7100 conf-sanity test_84 LBUGS with “(llog_osd.c:811:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index )”
• is related to: LU-7509 Failover: recovery-random-scale test_fail_client_mds: /dev/lvm-Role_MDS/P1 failed to initialize!
• is related to: LU-7364 conf-sanity test_84 fails with llog_process_thread()) invalid length 0 in llog record for index 0/68 'Restart of mds1 failed!'
• is related to: LU-6029 conf-sanity test_84: recovery_duration > recovery_time_hard
• is related to: LU-7169 conf-sanity 84 restart mds1 failed
• is related to: LU-7097 conf-sanity test_84 (check recovery_time_hard) fails on DNE setup

Comments

[Jian Yu]

Hi Hongchao,
Since the above failure is specific to dm-flakey patch, I filed LU-8729 to track it.

[Jian Yu]

Hi Hongchao,

With patch http://review.whamcloud.com/7200 on master branch, conf-sanity test 84 failed as follows:

CMD: onyx-31vm7 e2label /dev/mapper/mds1_flakey 				2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Update not seen after 90s: wanted '' got 'lustre:MDT0000'
 conf-sanity test_84: @@@@@@ FAIL: /dev/mapper/mds1_flakey failed to initialize! 

https://testing.hpdd.intel.com/test_sets/e88a61c2-89bf-11e6-a8b7-5254006e85c2

Could you please advise? Thank you.

[Peter Jones]

Test re-enabled for 2.9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20194/
Subject: LU-7428 test: remove test 84 from ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e40a3fd8a531ed60528ca82e02ce41918b1be6ba

[Andreas Dilger]

Reopen until patch enabling test_84 actually lands.

[Hongchao Zhang]

the patch http://review.whamcloud.com/#/c/20194/ to remove test 84 has been refreshed, and have passed the tests in Maloo now.

[John Hammond]

Did the landing of http://review.whamcloud.com/20586/ resolve this issue? I see that we still have 84 in ALWAYS_EXCEPT.

[Peter Jones]

Landed for 2.9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20586/
Subject: LU-7428 osd: set device read-only correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a079ade7913b923b795ea5c01df4e69bf1a87691

[Andreas Dilger]

No, this won't replace LU-684.

This patch is to (hopefully) fix a problem where the device is sync'd and set read-only, but loses some recent writes, for an unknown reason. This shows up with a variety of different symptoms, and may be a result of bad interactions with LVM and VM virtual block devices, or it may be caused by the dev readonly patches.