https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3264 Lustre <p>While running recovery-*-scale tests with FSTYPE=zfs and FAILURE_MODE=HARD under failover configuration, the tests failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Failing mds1 on wtm-9vm3 + pm -h powerman --off wtm-9vm3 Command completed successfully waiting ! ping -w 3 -c 1 wtm-9vm3, 4 secs left ... waiting ! ping -w 3 -c 1 wtm-9vm3, 3 secs left ... waiting ! ping -w 3 -c 1 wtm-9vm3, 2 secs left ... waiting ! ping -w 3 -c 1 wtm-9vm3, 1 secs left ... waiting for wtm-9vm3 to fail attempts=3 + pm -h powerman --off wtm-9vm3 Command completed successfully reboot facets: mds1 + pm -h powerman --on wtm-9vm3 Command completed successfully Failover mds1 to wtm-9vm7 04:28:49 (1367234929) waiting for wtm-9vm7 network 900 secs ... 04:28:49 (1367234929) network interface is UP CMD: wtm-9vm7 hostname mount facets: mds1 Starting mds1: lustre-mdt1/mdt1 /mnt/mds1 CMD: wtm-9vm7 mkdir -p /mnt/mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/mds1 wtm-9vm7: mount.lustre: lustre-mdt1/mdt1 has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool Start of lustre-mdt1/mdt1 on mds1 failed 19 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/ac7cbc10-b0e3-11e2-b2c4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ac7cbc10-b0e3-11e2-b2c4-52540035b04c</a></p> <br/> FSTYPE=zfs<br/> FAILURE_MODE=HARD<br/> LU-3264

recovery-*-scale tests failed with FSTYPE=zfs and FAILURE_MODE=HARD

Bug Critical Closed Fixed Jian Yu Jian Yu zfs Thu, 2 May 2013 10:20:10 +0000 Thu, 15 Aug 2013 06:30:11 +0000 Tue, 23 Jul 2013 18:25:32 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 0 12 <p>(CC'ed Brian. How does LLNL implement failovers with ZFS?)</p> <p>The pool lustre-mdt1 needs to be imported via "zpool import -f ..." on wtm-9vm7. The tricky part, however, is how to prevent wtm-9vm3 from playing with the pool after rebooting. It might be doable by never caching Lustre pool configurations ("-o cachefile=none" at creation time), so that none of them will be automatically imported anywhere. It would be great if two nodes and a shared device are available to experiments.</p> <blockquote><p>It would be great if two nodes and a shared device are available to experiments.</p></blockquote> <p>Let me setup the test environment and do some experiments.</p> <p>This kind of problem is why we would want to have MMP for ZFS, but that hasn't been developed yet. However, for the sake of this bug, we just need to fix the ZFS import problem so that our automated testing scripts work. </p> <p>Until we have MMP for ZFS we've resolved this issue by delegating full authority for starting/stopping servers to heartbeat. See the lustre/scripts/Lustre.ha_v2 resource scripts ZPOOL_IMPORT_ARGS='-f' line which is used to always force importing the pool. We also boot all of our nodes diskless so they never have a persistent cache file and thus never get automatically imported. I admit it's a stop gap until we have real MMP, but in practice it's been working thus far. </p> <p>Thanks, Brian. There's little info like this on the web. (Perhaps it would be worthwhile to add an FAQ entry on zfsonlinux.org sometime.)</p> <p>Patch for master branch is in <a href="http://review.whamcloud.com/6258" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6258</a>.</p> <p>Patch was landed on master branch.</p> <p>can you confirm the patch does work on a local setup?</p> <p>with REFORMAT=y FSTYPE=zfs sh llmount.sh -v I'm getting:</p> <p>Format mds1: lustre-mdt1/mdt1<br/> CMD: centos grep -c /mnt/mds1' ' /proc/mounts<br/> CMD: centos lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST '<br/> CMD: centos ! zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 ||<br/> zpool export lustre-mdt1<br/> CMD: centos /work/lustre/head1/lustre/tests/../utils/mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/work/lustre/head1/lustre/tests/../utils/l_getidentity --backfstype=zfs --device-size=200000 --reformat lustre-mdt1/mdt1 /tmp/lustre-mdt1</p> <p> Permanent disk data:<br/> Target: lustre:MDT0000<br/> Index: 0<br/> Lustre FS: lustre<br/> Mount type: zfs<br/> Flags: 0x65<br/> (MDT MGS first_time update )<br/> Persistent mount opts: <br/> Parameters: sys.timeout=20 lov.stripesize=1048576 lov.stripecount=0 mdt.identity_upcall=/work/lustre/head1/lustre/tests/../utils/l_getidentity</p> <p>mkfs_cmd = zpool create -f -O canmount=off lustre-mdt1 /tmp/lustre-mdt1<br/> mkfs_cmd = zfs create -o canmount=off -o xattr=sa lustre-mdt1/mdt1<br/> Writing lustre-mdt1/mdt1 properties<br/> lustre:version=1<br/> lustre:flags=101<br/> lustre:index=0<br/> lustre:fsname=lustre<br/> lustre:svname=lustre:MDT0000<br/> lustre:sys.timeout=20<br/> lustre:lov.stripesize=1048576<br/> lustre:lov.stripecount=0<br/> lustre:mdt.identity_upcall=/work/lustre/head1/lustre/tests/../utils/l_getidentity<br/> CMD: centos zpool set cachefile=none lustre-mdt1<br/> CMD: centos ! zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 ||<br/> zpool export lustre-mdt1<br/> ...<br/> Loading modules from /work/lustre/head1/lustre/tests/..<br/> detected 2 online CPUs by sysfs<br/> Force libcfs to create 2 CPU partitions<br/> debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super<br/> subsystem_debug=all -lnet -lnd -pinger<br/> gss/krb5 is not supported<br/> Setup mgs, mdt, osts<br/> CMD: centos mkdir -p /mnt/mds1<br/> CMD: centos zpool import -f -o cachefile=none lustre-mdt1<br/> cannot import 'lustre-mdt1': no such pool available</p> <blockquote><p>cannot import 'lustre-mdt1': no such pool available</p></blockquote> <p>For "zpool import" command, if the -d option is not specified, the command will only search for devices in "/dev". However, for ZFS storage pool which has file-based virtual device, we need explicitly specify the search directory otherwise the import command will not find the device.</p> <p>The patch for master branch is in <a href="http://review.whamcloud.com/6358" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6358</a>.</p> <blockquote><p>The patch for master branch is in <a href="http://review.whamcloud.com/6358" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6358</a>.</p></blockquote> <p>The patch was landed on both Lustre b2_4 and master branches.</p> <p>Reworked patch with fixes merged in:</p> <p><a href="http://review.whamcloud.com/6429" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6429</a></p> <p><a href="http://review.whamcloud.com/6429" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6429</a> merged</p> <blockquote><p><a href="http://review.whamcloud.com/6429" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6429</a> merged</p></blockquote> <p>The patch needs to be back-ported to Lustre b2_4 branch.</p> <p>Patch was landed on Lustre b2_4 branch.</p> Development Rank 1|hzvq0v: Rank (Obsolete) 8083 Severity