Lustre is not mounted, trying to do setup ... Stopping clients: client-4.lab.whamcloud.com /mnt/lustre (opts:) Stopping clients: client-4.lab.whamcloud.com /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff ../lnet/lnet/lnet options: 'accept=all' gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/sdb Format ost1: /dev/sdc1 Format ost2: /dev/sdc2 Checking servers environments Checking clients client-4.lab.whamcloud.com environments Loading modules from /usr/lib64/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o user_xattr,acl /dev/sdb /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-MDT0000 Starting ost1: /dev/sdc1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0000 Starting ost2: /dev/sdc2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0001 Starting client: client-4.lab.whamcloud.com: -o user_xattr,acl,flock fat-amd-4@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Using TIMEOUT=20 disable quota as required Logging to local directory: /home/yujian/test_logs/2011-03-08/235556 running: mmp ONLY=9 10 run_suite mmp /usr/lib64/lustre/tests/mmp.sh -----============= acceptance-small: mmp ============----- Tue Mar 8 23:56:22 PST 2011 Running: bash /usr/lib64/lustre/tests/mmp.sh Stopping clients: client-4.lab.whamcloud.com /mnt/lustre (opts:) sh: lsof: command not found Stopping client client-4.lab.whamcloud.com /mnt/lustre opts: Stopping clients: client-4.lab.whamcloud.com /mnt/lustre2 (opts:) Stopping /mnt/mds1 (opts:-f) Stopping /mnt/ost1 (opts:-f) Stopping /mnt/ost2 (opts:-f) modules unloaded. only running test 9 10 Loading modules from /usr/lib64/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff ../lnet/lnet/lnet options: 'accept=all' gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' excepting tests: Failover is not used on MDS, enabling MMP manually... tune2fs 1.41.14.wc1 (20-Jan-2011) Multiple mount protection has been enabled with update interval 1s. Failover is not used on OSS, enabling MMP manually... tune2fs 1.41.14.wc1 (20-Jan-2011) Multiple mount protection has been enabled with update interval 1s. fat-amd-4: dumpe2fs 1.41.14.wc1 (20-Jan-2011) Filesystem features: has_journal ext_attr resize_inode dir_index filetype mmp dirdata sparse_super large_file uninit_bg fat-amd-4: dumpe2fs 1.41.14.wc1 (20-Jan-2011) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent mmp sparse_super large_file uninit_bg ........ == mmp test 9: mount after aborted e2fsck ============================================================ 23:56:45 (1299657405) Starting mds1: -o user_xattr,acl /dev/sdb /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-MDT0000 Starting ost1: /dev/sdc1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0000 Stopping /mnt/mds1 (opts:) Stopping /mnt/ost1 (opts:) Running e2fsck on the device /dev/sdb on mds1... fat-amd-4: e2fsck 1.41.14.wc1 (20-Jan-2011) Mounting /dev/sdb on mds1... /usr/lib64/lustre/tests/test-framework.sh: line 267: 24391 Aborted run_e2fsck $MMP_MDS_FAILOVER $MMP_MDSDEV "-fy" Starting mds1: -o user_xattr,acl /dev/sdb /mnt/mds1 fat-amd-4: mount.lustre: mount /dev/sdb at /mnt/mds1 failed: Device or resource busy mount -t lustre /dev/sdb /mnt/mds1 Start of /dev/sdb on mds1 failed 16 tune2fs 1.41.14.wc1 (20-Jan-2011) Running e2fsck on the device /dev/sdc1 on ost1... fat-amd-4: e2fsck 1.41.14.wc1 (20-Jan-2011) fat-amd-4: e2fsck: MMP: device currently active while trying to open /dev/sdb The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 32768 Mounting /dev/sdc1 on ost1... /usr/lib64/lustre/tests/test-framework.sh: line 267: 24570 Aborted run_e2fsck $MMP_OSS_FAILOVER $MMP_OSTDEV "-fy" Starting ost1: /dev/sdc1 /mnt/ost1 fat-amd-4: mount.lustre: mount /dev/sdc1 at /mnt/ost1 failed: Device or resource busy mount -t lustre /dev/sdc1 /mnt/ost1 Start of /dev/sdc1 on ost1 failed 16 tune2fs 1.41.14.wc1 (20-Jan-2011) Resetting fail_loc on all nodes...done. fat-amd-4: e2fsck: MMP: device currently active while trying to open /dev/sdc1 fat-amd-4: bash: line 1: 22433 Segmentation fault LUSTRE="/usr/lib64/lustre" sh -c "e2fsck -fy /dev/sdc1" The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 PASS 9 (33s) == mmp test 10: e2fsck with mounted filesystem ======================================================= 23:57:18 (1299657438) Mounting /dev/sdb on mds1... Starting mds1: -o user_xattr,acl /dev/sdb /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-MDT0000 Running e2fsck on the device /dev/sdb on mds1... fat-amd-4: e2fsck 1.41.14.wc1 (20-Jan-2011) Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information lustre-MDT0000: 38/250048 files (7.9% non-contiguous), 35732/250000 blocks Mounting /dev/sdc1 on ost1... Starting ost1: /dev/sdc1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0000 Running e2fsck on the device /dev/sdc1 on ost1... fat-amd-4: e2fsck 1.41.14.wc1 (20-Jan-2011) fat-amd-4: device /dev/sdc1 mounted by lustre per /proc/fs/lustre/obdfilter/lustre-OST0000/mntdev Warning! /dev/sdc1 is mounted. Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information lustre-OST0000: 86/625856 files (1.2% non-contiguous), 144158/2500000 blocks Stopping /mnt/mds1 (opts:) Stopping /mnt/ost1 (opts:) Resetting fail_loc on all nodes...done. PASS 10 (12s) Failover is not used on MDS, disabling MMP manually... tune2fs 1.41.14.wc1 (20-Jan-2011) fat-amd-4: dumpe2fs 1.41.14.wc1 (20-Jan-2011) Failover is not used on OSS, disabling MMP manually... tune2fs 1.41.14.wc1 (20-Jan-2011) fat-amd-4: dumpe2fs 1.41.14.wc1 (20-Jan-2011) == mmp mmp.sh test complete, duration 91 sec ========================================================= 23:57:53 (1299657473) Checking servers environments Checking clients client-4.lab.whamcloud.com environments Loading modules from /usr/lib64/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o user_xattr,acl /dev/sdb /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-MDT0000 Starting ost1: /dev/sdc1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0000 Starting ost2: /dev/sdc2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Started lustre-OST0001 Starting client: client-4.lab.whamcloud.com: -o user_xattr,acl,flock fat-amd-4@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=24 Using TIMEOUT=20 disable quota as required mmp returned 0 Uploading /home/yujian/test_logs/2011-03-08/235556 to https://maloo.whamcloud.com/import_tasks /home/yujian/test_logs/2011-03-08/235556 /usr/lib64/lustre/tests % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 12009 0 1300 100 10709 6587 54262 --:--:-- --:--:-- --:--:-- 72850 /usr/lib64/lustre/tests Complete. Stopping clients: client-4.lab.whamcloud.com /mnt/lustre (opts:) sh: lsof: command not found Stopping client client-4.lab.whamcloud.com /mnt/lustre opts: Stopping clients: client-4.lab.whamcloud.com /mnt/lustre2 (opts:) Stopping /mnt/mds1 (opts:-f) Stopping /mnt/ost1 (opts:-f) Stopping /mnt/ost2 (opts:-f) modules unloaded. Finished at Tue Mar 8 23:58:33 PST 2011 in 157s auster: completed with rc 0