-----============= acceptance-small: replay-dual ============----- Tue Nov 24 15:23:09 PST 2015 Running: bash /usr/lib64/lustre/tests/replay-dual.sh excepting tests: 14b 15c skipping tests SLOW=no: 21b Starting client eagle-52vm5: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Started clients eagle-52vm5: eagle-52vm2@tcp:/lustre on /mnt/lustre2 type lustre (rw,user_xattr,flock) eagle-52vm5: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients eagle-52vm5 environments Using TIMEOUT=20 disable quota as required osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client == 15:23:34 (1448407414) Check file is LU482_FAILED=/tmp/replay-dual.lu482.uthzDO Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre total: 50 creates in 0.46 seconds: 109.12 creates/second fail_loc=0x80000514 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:23:53 (1448407433) waiting for eagle-52vm2 network 900 secs ... 15:23:53 (1448407433) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1448407509 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second Resetting fail_loc on all nodes...done. PASS 0a (96s) == replay-dual test 0b: lost client during waiting for next transno == 15:25:10 (1448407510) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:25:28 (1448407528) waiting for eagle-52vm2 network 900 secs ... 15:25:28 (1448407528) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Resetting fail_loc on all nodes...done. PASS 0b (86s) == replay-dual test 1: |X| simple create == 15:26:36 (1448407596) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:26:54 (1448407614) waiting for eagle-52vm2 network 900 secs ... 15:26:54 (1448407614) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec Resetting fail_loc on all nodes...done. PASS 1 (25s) == replay-dual test 2: |X| mkdir adir == 15:27:01 (1448407621) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:27:19 (1448407639) waiting for eagle-52vm2 network 900 secs ... 15:27:19 (1448407639) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Resetting fail_loc on all nodes...done. PASS 2 (31s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir == 15:27:32 (1448407652) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:27:50 (1448407670) waiting for eagle-52vm2 network 900 secs ... 15:27:50 (1448407670) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec Resetting fail_loc on all nodes...done. PASS 3 (23s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir == 15:27:55 (1448407675) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre mkdir: cannot create directory `/mnt/lustre/adir': File exists Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:28:13 (1448407693) waiting for eagle-52vm2 network 900 secs ... 15:28:13 (1448407693) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec Resetting fail_loc on all nodes...done. PASS 4 (27s) == replay-dual test 5: open, unlink |X| close == 15:28:22 (1448407702) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.23363 Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:28:40 (1448407720) waiting for eagle-52vm2 network 900 secs ... 15:28:40 (1448407720) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec Resetting fail_loc on all nodes...done. PASS 5 (27s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 == 15:28:49 (1448407729) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.23363 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.23363 Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:29:07 (1448407747) waiting for eagle-52vm2 network 900 secs ... 15:29:07 (1448407747) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec Resetting fail_loc on all nodes...done. PASS 6 (24s) == replay-dual test 8: replay of resent request == 15:29:13 (1448407753) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:29:38 (1448407778) waiting for eagle-52vm2 network 900 secs ... 15:29:38 (1448407778) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec Resetting fail_loc on all nodes...done. PASS 8 (39s) == replay-dual test 9: resending a replayed create == 15:29:52 (1448407792) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x80000119 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:30:10 (1448407810) waiting for eagle-52vm2 network 900 secs ... 15:30:10 (1448407810) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 12 sec fail_loc=0 Resetting fail_loc on all nodes...done. PASS 9 (34s) == replay-dual test 10: resending a replayed unlink == 15:30:26 (1448407826) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x80000119 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:30:44 (1448407844) waiting for eagle-52vm2 network 900 secs ... 15:30:44 (1448407844) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec fail_loc=0 Resetting fail_loc on all nodes...done. PASS 10 (37s) == replay-dual test 11: both clients timeout during replay == 15:31:04 (1448407864) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x0119 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:31:22 (1448407882) waiting for eagle-52vm2 network 900 secs ... 15:31:22 (1448407882) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec fail_loc=0 Resetting fail_loc on all nodes...done. PASS 11 (37s) == replay-dual test 12: open resend timeout == 15:31:40 (1448407900) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.23363 fail_loc=0x80000302 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:31:58 (1448407918) waiting for eagle-52vm2 network 900 secs ... 15:31:58 (1448407918) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK Resetting fail_loc on all nodes...done. PASS 12 (28s) == replay-dual test 13: close resend timeout == 15:32:08 (1448407928) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.23363 Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x80000115 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:32:26 (1448407946) waiting for eagle-52vm2 network 900 secs ... 15:32:26 (1448407946) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK Resetting fail_loc on all nodes...done. PASS 13 (27s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes == 15:32:36 (1448407956) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre total: 25 creates in 0.25 seconds: 99.74 creates/second total: 1 creates in 0.01 seconds: 117.32 creates/second Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:32:54 (1448407974) waiting for eagle-52vm2 network 900 secs ... 15:32:54 (1448407974) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 68 sec - unlinked 0 (time 1448408046 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Resetting fail_loc on all nodes...done. PASS 15a (91s) SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c == replay-dual test 16: fail MDS during recovery (3571) == 15:34:07 (1448408047) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre total: 25 creates in 0.22 seconds: 113.44 creates/second total: 1 creates in 0.01 seconds: 102.68 creates/second Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:34:25 (1448408065) waiting for eagle-52vm2 network 900 secs ... 15:34:25 (1448408065) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:35:04 (1448408104) waiting for eagle-52vm2 network 900 secs ... 15:35:04 (1448408104) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 77 sec - unlinked 0 (time 1448408185 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Resetting fail_loc on all nodes...done. PASS 16 (138s) == replay-dual test 17: fail OST during recovery (3571) == 15:36:25 (1448408185) total: 25 creates in 0.24 seconds: 105.10 creates/second total: 1 creates in 0.01 seconds: 82.86 creates/second Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing ost1 on eagle-52vm3 Stopping /mnt/ost0 (opts:) on eagle-52vm3 reboot facets: ost1 Failover ost1 to eagle-52vm3 15:36:37 (1448408197) waiting for eagle-52vm3 network 900 secs ... 15:36:37 (1448408197) network interface is UP mount facets: ost1 Starting ost1: /dev/vda5 /mnt/ost0 Started lustre-OST0000 Failing ost1 on eagle-52vm3 Stopping /mnt/ost0 (opts:) on eagle-52vm3 reboot facets: ost1 Failover ost1 to eagle-52vm3 15:37:10 (1448408230) waiting for eagle-52vm3 network 900 secs ... 15:37:10 (1448408230) network interface is UP mount facets: ost1 Starting ost1: /dev/vda5 /mnt/ost0 Started lustre-OST0000 eagle-52vm5: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 69 sec - unlinked 0 (time 1448408303 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Resetting fail_loc on all nodes...done. PASS 17 (118s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) == 15:38:23 (1448408303) debug=+dlmtrace fail_loc=0x8000030b using seed 1886656280 running for 500 iterations - stat 0 (time 1448408304 ; total 0 ; last 0) total: 500 stats in 0 seconds: inf stats/second ldlm.namespaces.MGC10.100.4.183@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88006018ac00.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88007b0ca800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88006018ac00.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88007b0ca800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88006018ac00.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88007b0ca800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC10.100.4.183@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88006018ac00.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88007b0ca800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88006018ac00.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88007b0ca800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88006018ac00.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88007b0ca800.early_lock_cancel=1 fail_loc=0 fail_loc=0 Resetting fail_loc on all nodes...done. PASS 18 (44s) == replay-dual test 19: resend of open request == 15:39:07 (1448408347) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre fail_loc=0x30c fail_loc=0x30c total: 1 creates in 0.01 seconds: 95.48 creates/second fail_loc=0 fail_loc=0 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:39:25 (1448408365) waiting for eagle-52vm2 network 900 secs ... 15:39:25 (1448408365) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Resetting fail_loc on all nodes...done. PASS 19 (23s) == replay-dual test 20: recovery time is not increasing == 15:39:30 (1448408370) Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:39:49 (1448408389) waiting for eagle-52vm2 network 900 secs ... 15:39:49 (1448408389) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 59736 649672 9% /mnt/lustre Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:42:26 (1448408546) waiting for eagle-52vm2 network 900 secs ... 15:42:26 (1448408546) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 eagle-52vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 Resetting fail_loc on all nodes...done. PASS 20 (315s) == replay-dual test 21a: commit on sharing == 15:44:45 (1448408685) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 reboot facets: mds1 Failover mds1 to eagle-52vm2 15:45:03 (1448408703) waiting for eagle-52vm2 network 900 secs ... 15:45:03 (1448408703) network interface is UP mount facets: mds1 Starting mds1: /dev/vda5 /mnt/mds0 Started lustre-MDT0000 Starting client: eagle-52vm5.eagle.hpdd.intel.com: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 Resetting fail_loc on all nodes...done. PASS 21a (156s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir == 15:47:21 (1448408841) SKIP: replay-dual test_22a needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 22a (1s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir == 15:47:22 (1448408842) SKIP: replay-dual test_22b needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 22b (1s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir == 15:47:23 (1448408843) SKIP: replay-dual test_22c needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir == 15:47:24 (1448408844) SKIP: replay-dual test_22d needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 22d (1s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 == 15:47:25 (1448408845) SKIP: replay-dual test_23a needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 23a (1s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 == 15:47:26 (1448408846) SKIP: replay-dual test_23b needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 == 15:47:27 (1448408847) SKIP: replay-dual test_23c needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 23c (0s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 == 15:47:28 (1448408848) SKIP: replay-dual test_23d needs >= 2 MDTs Resetting fail_loc on all nodes...done. SKIP 23d (0s) == replay-dual test 24: reconstruct on non-existing object == 15:47:28 (1448408848) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory Resetting fail_loc on all nodes...done. PASS 24 (77s) == replay-dual test 25: replay|resend == 15:48:45 (1448408925) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00320076 s, 160 kB/s fail_loc=0x304 fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on eagle-52vm3 Stopping /mnt/ost0 (opts:) on eagle-52vm3 reboot facets: ost1 Failover ost1 to eagle-52vm3 15:48:58 (1448408938) waiting for eagle-52vm3 network 900 secs ... 15:48:58 (1448408938) network interface is UP mount facets: ost1 Starting ost1: /dev/vda5 /mnt/ost0 Started lustre-OST0000 eagle-52vm5: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec /usr/lib64/lustre/tests/test-framework.sh: line 4421: 21912 Terminated LUSTRE="/usr/lib64/lustre" sh -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 fail_loc=0 Resetting fail_loc on all nodes...done. PASS 25 (20s) == replay-dual test 26: dbench and tar with mds failover == 15:49:05 (1448408945) Starting client eagle-52vm5: -o user_xattr,flock eagle-52vm2@tcp:/lustre /mnt/lustre Started clients eagle-52vm5: eagle-52vm2@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,flock) Started tar 23149 Started dbench 23150 tar: Removing leading `/' from member names Filesystem 1K-blocks Used Available Use% Mounted on eagle-52vm2@tcp:/lustre 749872 86376 614872 13% /mnt/lustre test_26 fail mds1 1 times Failing mds1 on eagle-52vm2 Stopping /mnt/mds0 (opts:) on eagle-52vm2 tar: Removing leading `/' from member names tar: Removing leading `/' from member names tar: Removing leading `/' from member names