[LU-3264] recovery-*-scale tests failed with FSTYPE=zfs and FAILURE_MODE=HARD

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.4.1, Lustre 2.5.0
• Affects Version/s: Lustre 2.4.0
• Labels: zfs
• Environment: FSTYPE=zfs; FAILURE_MODE=HARD
• Severity: 3
• Assignee: Jian Yu
• Reporter: Jian Yu

Description

While running recovery-*-scale tests with FSTYPE=zfs and FAILURE_MODE=HARD under failover configuration, the tests failed as follows:

Failing mds1 on wtm-9vm3
+ pm -h powerman --off wtm-9vm3
Command completed successfully
waiting ! ping -w 3 -c 1 wtm-9vm3, 4 secs left ...
waiting ! ping -w 3 -c 1 wtm-9vm3, 3 secs left ...
waiting ! ping -w 3 -c 1 wtm-9vm3, 2 secs left ...
waiting ! ping -w 3 -c 1 wtm-9vm3, 1 secs left ...
waiting for wtm-9vm3 to fail attempts=3
+ pm -h powerman --off wtm-9vm3
Command completed successfully
reboot facets: mds1
+ pm -h powerman --on wtm-9vm3
Command completed successfully
Failover mds1 to wtm-9vm7
04:28:49 (1367234929) waiting for wtm-9vm7 network 900 secs ...
04:28:49 (1367234929) network interface is UP
CMD: wtm-9vm7 hostname
mount facets: mds1
Starting mds1:   lustre-mdt1/mdt1 /mnt/mds1
CMD: wtm-9vm7 mkdir -p /mnt/mds1; mount -t lustre   		                   lustre-mdt1/mdt1 /mnt/mds1
wtm-9vm7: mount.lustre: lustre-mdt1/mdt1 has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool
Start of lustre-mdt1/mdt1 on mds1 failed 19

Maloo report: https://maloo.whamcloud.com/test_sets/ac7cbc10-b0e3-11e2-b2c4-52540035b04c

Comments

[Alex Zhuravlev]

can you confirm the patch does work on a local setup?

[Jian Yu]

Patch was landed on master branch.

[Jian Yu]

Patch for master branch is in http://review.whamcloud.com/6258.

[Li Wei (Inactive)]

Thanks, Brian. There's little info like this on the web. (Perhaps it would be worthwhile to add an FAQ entry on zfsonlinux.org sometime.)

[Brian Behlendorf]

Until we have MMP for ZFS we've resolved this issue by delegating full authority for starting/stopping servers to heartbeat. See the lustre/scripts/Lustre.ha_v2 resource scripts ZPOOL_IMPORT_ARGS='-f' line which is used to always force importing the pool. We also boot all of our nodes diskless so they never have a persistent cache file and thus never get automatically imported. I admit it's a stop gap until we have real MMP, but in practice it's been working thus far.

[Andreas Dilger]

This kind of problem is why we would want to have MMP for ZFS, but that hasn't been developed yet. However, for the sake of this bug, we just need to fix the ZFS import problem so that our automated testing scripts work.

[Jian Yu]

It would be great if two nodes and a shared device are available to experiments.

Let me setup the test environment and do some experiments.

[Li Wei (Inactive)]

(CC'ed Brian. How does LLNL implement failovers with ZFS?)

The pool lustre-mdt1 needs to be imported via "zpool import -f ..." on wtm-9vm7. The tricky part, however, is how to prevent wtm-9vm3 from playing with the pool after rebooting. It might be doable by never caching Lustre pool configurations ("-o cachefile=none" at creation time), so that none of them will be automatically imported anywhere. It would be great if two nodes and a shared device are available to experiments.