-----============= acceptance-small: recovery-mds-scale ============----- Fri Jul 31 17:25:52 PDT 2015 Exit config file opensfs.sh c03: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c24: Exit config file opensfs.sh c04: Exit config file opensfs.sh c06: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh mds02: Exit config file opensfs.sh oss01: Exit config file opensfs.sh oss02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh excepting tests: c01: Exit config file opensfs.sh c01: Checking config lustre mounted on /lustre/lustre c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c06: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Checking config lustre mounted on /lustre/lustre c07: Checking config lustre mounted on /lustre/lustre c02: Checking config lustre mounted on /lustre/lustre c06: Checking config lustre mounted on /lustre/lustre c05: Checking config lustre mounted on /lustre/lustre c03: Exit config file opensfs.sh c24: Exit config file opensfs.sh c04: Exit config file opensfs.sh c03: Checking config lustre mounted on /lustre/lustre c24: Checking config lustre mounted on /lustre/lustre c04: Checking config lustre mounted on /lustre/lustre Checking servers environments Checking clients c01,c02,c03,c04,c05,c06,c07,c08,c24 environments Using TIMEOUT=100 enable quota as required [HOST:c24] [old_mdt_qtype:none] [old_ost_qtype:none] [new_qtype:] mds01: warning: 'lctl conf_param' is deprecated, use 'lctl set_param -P' instead mds01: warning: 'lctl conf_param' is deprecated, use 'lctl set_param -P' instead Total disk size: 83624752 block-softlimit: 83625776 block-hardlimit: 87807064 inode-softlimit: 1235598 inode-hardlimit: 1297377 c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c03: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c06: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh oss01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh oss02: Exit config file opensfs.sh osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 Stopping client c24 /lustre/lustre (opts:) c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c06: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c03: Exit config file opensfs.sh == recovery-mds-scale test failover_mds: failover MDS == 17:26:02 (1438388762) Started client load: tar on c01 Started client load: dbench on c02 Started client load: dd on c03 Started client load: tar on c04 Started client load: dbench on c05 Started client load: dd on c06 Started client load: tar on c07 Started client load: dbench on c08 client loads pids: c06: 9211 c03: 9137 c02: 9113 c08: 9241 c07: 9230 c05: 9185 c01: 9114 c04: 9141 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 17:26:23 (1438388783) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 17:29:27 (1438388967) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 1 times, and counting... sleeping 1585 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=215 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 17:56:24 (1438390584) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 17:59:22 (1438390762) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 2 times, and counting... sleeping 1591 seconds... striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 ststriped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 ststriped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 st==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=2011 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 18:26:25 (1438392385) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 18:29:29 (1438392569) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 1 times, and counting... sleeping 1582 seconds... riped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 stririped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 stririped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 stri==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=3820 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 18:56:42 (1438394202) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 18:59:41 (1438394381) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 1 times, and counting... sleeping 1571 seconds... striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 sstriped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 s==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=5631 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 19:26:25 (1438395985) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 19:29:29 (1438396169) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 2 times, and counting... sleeping 1585 seconds... ped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 stripeped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 stripeped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 stripe==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=7418 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 19:56:43 (1438397803) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 19:59:39 (1438397979) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 2 times, and counting... sleeping 1575 seconds... d dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped d dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped d dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=9228 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 20:26:26 (1438399586) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... 20:29:24 (1438399764) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 3 times, and counting... sleeping 1590 seconds... triped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 sttriped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stdir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped didir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped didir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped di==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=11013 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 20:56:26 (1438401386) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... waiting ping -c 1 -w 3 mds04, 785 secs left ... 20:59:30 (1438401570) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 1 times, and counting... sleeping 1580 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=12823 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 21:26:26 (1438403186) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 21:29:24 (1438403364) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 2 times, and counting... sleeping 1591 seconds... r -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir r -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir r -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=14612 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 21:56:26 (1438404986) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 21:59:30 (1438405170) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 4 times, and counting... sleeping 1585 seconds... riped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 strriped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 str-i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i-i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i-i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=16419 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c06: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 22:26:26 (1438406786) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 22:29:30 (1438406970) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 3 times, and counting... sleeping 1585 seconds... 1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=18219 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 22:56:44 (1438408604) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 22:59:40 (1438408780) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 3 times, and counting... sleeping 1575 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=20030 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c03: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 23:26:29 (1438410389) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 23:29:26 (1438410566) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 4 times, and counting... sleeping 1587 seconds... -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c-c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c-c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -ciped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striiped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stri==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=21818 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 23:56:29 (1438412189) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 23:59:25 (1438412365) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 5 times, and counting... sleeping 1590 seconds... 4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=23615 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c03: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 00:26:28 (1438413988) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 00:29:26 (1438414166) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 3 times, and counting... sleeping 1587 seconds... /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /l/lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /l/lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /l==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=25419 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 00:56:29 (1438415789) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 00:59:27 (1438415967) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 6 times, and counting... sleeping 1588 seconds... ped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 strip==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=27218 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 01:26:46 (1438417606) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 01:29:43 (1438417783) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 4 times, and counting... sleeping 1573 seconds... ustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lusustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lusustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lus==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=29033 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 01:56:29 (1438419389) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 01:59:33 (1438419573) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 5 times, and counting... sleeping 1584 seconds... tre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustrtre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustrtre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustr==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=30822 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 02:26:30 (1438421190) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 02:29:34 (1438421374) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 6 times, and counting... sleeping 1584 seconds... ed dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripeed dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stripee/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/e/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=32623 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 02:56:30 (1438422990) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 02:59:34 (1438423174) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 7 times, and counting... sleeping 1584 seconds... e/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=34423 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 03:26:30 (1438424790) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 03:29:34 (1438424974) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 8 times, and counting... sleeping 1579 seconds... lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lulustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lulustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lu==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=36228 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 03:56:30 (1438426590) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 03:59:34 (1438426774) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 9 times, and counting... sleeping 1580 seconds... d dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripedstre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/luststre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/luststre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lust==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=38027 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 04:26:35 (1438428395) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 04:29:32 (1438428572) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 5 times, and counting... sleeping 1585 seconds... d dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=39823 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 04:56:31 (1438430191) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 04:59:29 (1438430369) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 7 times, and counting... sleeping 1586 seconds... re/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustrere/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustrere/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=41622 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 05:26:31 (1438431991) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 05:29:28 (1438432168) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 8 times, and counting... sleeping 1591 seconds... /d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=43418 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 05:56:31 (1438433791) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 05:59:29 (1438433969) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 9 times, and counting... sleeping 1587 seconds... dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped 0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=45222 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 06:26:31 (1438435591) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 06:29:35 (1438435775) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 10 times, and counting... sleeping 1582 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=47027 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 06:56:32 (1438437392) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 06:59:30 (1438437570) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 10 times, and counting... sleeping 1591 seconds... tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tatar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tatar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tadir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped d==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=48818 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 07:26:32 (1438439192) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... waiting ping -c 1 -w 3 mds04, 785 secs left ... 07:29:36 (1438439376) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 4 times, and counting... sleeping 1581 seconds... dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dr-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-r-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-r-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=50628 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 07:56:32 (1438440992) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 07:59:36 (1438441176) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 11 times, and counting... sleeping 1585 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=52425 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 08:26:33 (1438442793) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 08:29:31 (1438442971) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 5 times, and counting... sleeping 1588 seconds... c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c0c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c0c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c0ir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped di==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=54222 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 08:56:33 (1438444593) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... 08:59:31 (1438444771) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 11 times, and counting... sleeping 1589 seconds... ir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped di4 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 1 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 7 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=56022 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 09:26:51 (1438446411) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 09:29:47 (1438446587) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 6 times, and counting... sleeping 1572 seconds... striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 ststriped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 ststriped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 st==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=57839 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 09:56:34 (1438448194) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 09:59:38 (1438448378) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 12 times, and counting... sleeping 1579 seconds... striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lstriped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lstriped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lr -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=59632 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 10:26:34 (1438449994) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 10:29:32 (1438450172) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 12 times, and counting... sleeping 1585 seconds... riped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 stririped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 stririped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 strir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=61426 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 10:56:39 (1438451799) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 10:59:35 (1438451975) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 7 times, and counting... sleeping 1588 seconds... ped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 stripeped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 stripeped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 stripe==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=63224 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c03: Exit config file opensfs.sh c01: Exit config file opensfs.sh c06: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 11:26:34 (1438453594) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 11:29:32 (1438453772) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 6 times, and counting... sleeping 1586 seconds... -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=65026 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 11:56:35 (1438455395) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 11:59:39 (1438455579) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 13 times, and counting... sleeping 1580 seconds... d dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped d dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped d dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=66832 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c03: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 12:26:36 (1438457196) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 12:29:34 (1438457374) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 7 times, and counting... sleeping 1586 seconds... dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped didir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped didir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped di==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=68627 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 12:56:36 (1438458996) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 12:59:34 (1438459174) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 13 times, and counting... sleeping 1586 seconds... r -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir r -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir r -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=70427 DURATION=86400 PERIOD=1800 -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 13:26:36 (1438460796) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 13:29:40 (1438460980) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 14 times, and counting... sleeping 1580 seconds... -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=72233 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 13:56:37 (1438462597) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 13:59:41 (1438462781) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 15 times, and counting... sleeping 1584 seconds... -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i-i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i-i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=74030 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 14:26:37 (1438464397) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 14:29:35 (1438464575) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 14 times, and counting... sleeping 1588 seconds... 0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=75826 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 14:56:38 (1438466198) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 14:59:36 (1438466376) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 15 times, and counting... sleeping 1591 seconds... i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i-c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c-c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c-c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=77624 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 15:26:38 (1438467998) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 15:29:43 (1438468183) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 16 times, and counting... sleeping 1580 seconds... i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=79436 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 15:56:39 (1438469799) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 15:59:43 (1438469983) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 16 times, and counting... sleeping 1583 seconds... 4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=81233 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 16:26:39 (1438471599) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 16:29:36 (1438471776) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 8 times, and counting... sleeping 1589 seconds... 2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2/lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /l/lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /l/lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /l==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=83028 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 16:56:41 (1438473401) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 16:59:39 (1438473579) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 9 times, and counting... 2015-08-01 16:59:58 Terminating clients loads ... Duration: 86400 Server failover period: 1800 seconds Exited after: 84832 seconds Number of failovers before exit: mds1: 16 times mds2: 7 times mds3: 16 times mds4: 9 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times Status: PASS: rc=0 PASS failover_mds (84837s)