== recovery-mds-scale test failover_mds: failover MDS == 00:09:06 (1437980946) Started client load: tar on c01 Started client load: dbench on c02 Started client load: dd on c03 Started client load: tar on c04 Started client load: dbench on c05 Started client load: dd on c06 Started client load: tar on c07 Started client load: dbench on c08 client loads pids: c03: 9162 c06: 9223 c08: 9282 c01: 9126 c07: 9258 c02: 9138 c04: 9165 c05: 9200 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 00:09:27 (1437980967) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 00:12:31 (1437981151) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 1 times, and counting... sleeping 1586 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=214 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 00:39:30 (1437982770) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 00:42:34 (1437982954) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 2 times, and counting... sleeping 1581 seconds... striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 ststriped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 ststriped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 st==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=2021 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 01:09:30 (1437984570) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 01:12:28 (1437984748) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 1 times, and counting... sleeping 1589 seconds... riped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 stririped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 stririped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 stri==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=3813 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 01:39:30 (1437986370) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 01:42:34 (1437986554) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 3 times, and counting... sleeping 1580 seconds... striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 sstriped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 s==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=5622 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 02:09:31 (1437988171) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... 02:12:30 (1437988350) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 4 times, and counting... sleeping 1589 seconds... ped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 stripeped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 stripeped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 stripe==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=7414 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 02:39:31 (1437989971) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... waiting ping -c 1 -w 3 mds04, 785 secs left ... 02:42:35 (1437990155) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 1 times, and counting... sleeping 1582 seconds... d dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped d dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped d dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=9222 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 03:09:34 (1437991774) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 03:12:33 (1437991953) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 2 times, and counting... sleeping 1583 seconds... triped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 sttriped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stdir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped didir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped didir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped di==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=11022 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 03:39:33 (1437993573) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... waiting ping -c 1 -w 3 mds04, 785 secs left ... 03:42:37 (1437993757) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 3 times, and counting... sleeping 1583 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=12822 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 04:09:33 (1437995373) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... waiting ping -c 1 -w 3 mds04, 785 secs left ... 04:12:37 (1437995557) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 4 times, and counting... sleeping 1584 seconds... r -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir r -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir r -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=14622 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c02: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 04:39:34 (1437997174) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 04:42:38 (1437997358) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 2 times, and counting... sleeping 1573 seconds... riped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 str-i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i-i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -iriped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 str-i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=16433 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 05:09:34 (1437998974) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 05:12:30 (1437999150) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 5 times, and counting... sleeping 1589 seconds... 3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=18217 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: INACTIVE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 05:39:52 (1438000792) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... waiting ping -c 1 -w 3 mds02, 785 secs left ... 05:42:56 (1438000976) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 1 times, and counting... sleeping 1566 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=20040 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 06:09:34 (1438002574) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 06:12:38 (1438002758) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 5 times, and counting... sleeping 1581 seconds... -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c-c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c-c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -ciped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striiped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stri==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=21825 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 06:39:51 (1438004391) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 06:42:47 (1438004567) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 2 times, and counting... sleeping 1574 seconds... 4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=23632 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 07:09:34 (1438006174) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 07:12:32 (1438006352) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 6 times, and counting... sleeping 1590 seconds... /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /l/lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /l/lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /l==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=25416 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 07:39:33 (1438007973) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 07:42:32 (1438008152) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 3 times, and counting... sleeping 1586 seconds... ped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 strip==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=27220 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 08:09:36 (1438009776) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 08:12:34 (1438009954) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 4 times, and counting... sleeping 1591 seconds... ustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lusustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lusustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lus==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=29017 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 08:39:36 (1438011576) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 08:42:40 (1438011760) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 6 times, and counting... sleeping 1583 seconds... tre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustrtre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustrtre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustr==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=30826 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 09:09:36 (1438013376) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 09:12:41 (1438013561) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 7 times, and counting... sleeping 1582 seconds... ed dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripeed dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 stripe==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=32627 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 09:39:43 (1438015183) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 09:42:39 (1438015359) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 3 times, and counting... sleeping 1586 seconds... e/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/e/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/e/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=34424 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 10:09:38 (1438016978) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 10:12:35 (1438017155) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 5 times, and counting... sleeping 1591 seconds... lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lulustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lulustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lu==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=36219 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 10:39:38 (1438018778) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 10:42:36 (1438018956) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 7 times, and counting... sleeping 1591 seconds... d dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 stripedstre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/luststre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/luststre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lust==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=38020 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 11:09:56 (1438020596) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 11:12:53 (1438020773) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 4 times, and counting... sleeping 1571 seconds... d dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=39840 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 11:39:39 (1438022379) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 11:42:37 (1438022557) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 8 times, and counting... sleeping 1586 seconds... re/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustrere/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustrere/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=41625 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 12:09:39 (1438024179) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 12:12:35 (1438024355) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 6 times, and counting... sleeping 1588 seconds... /d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=43423 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 12:39:56 (1438025996) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 12:42:52 (1438026172) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 5 times, and counting... sleeping 1574 seconds... dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped 0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=45237 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 13:09:39 (1438027779) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 13:12:36 (1438027956) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 9 times, and counting... sleeping 1587 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=47024 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 13:39:56 (1438029596) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 13:42:52 (1438029772) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 6 times, and counting... sleeping 1574 seconds... tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tatar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tatar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.ta==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=48837 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 14:09:39 (1438031379) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 14:12:43 (1438031563) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 7 times, and counting... sleeping 1584 seconds... dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped ddir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dr-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-r-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-r-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=50627 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 14:39:39 (1438033179) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 14:42:43 (1438033363) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 8 times, and counting... sleeping 1580 seconds... c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c0c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c0c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c0==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=52431 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 15:09:38 (1438034978) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... waiting ping -c 1 -w 3 mds03, 785 secs left ... 15:12:43 (1438035163) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 8 times, and counting... sleeping 1579 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=54232 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 15:39:56 (1438036796) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 15:42:53 (1438036973) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 7 times, and counting... sleeping 1570 seconds... ir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped di1 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 4 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 7 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 ir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped di==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=56041 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: Exit config file opensfs.sh c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 16:09:44 (1438038584) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 16:12:41 (1438038761) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 8 times, and counting... sleeping 1582 seconds... striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 ststriped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 ststriped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 st==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=57829 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 16:39:56 (1438040396) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 16:42:52 (1438040572) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 9 times, and counting... sleeping 1574 seconds... striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c05 striped dir -i1 -c4 /lustre/lstriped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c02 striped dir -i1 -c4 /lustre/lstriped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i1 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i2 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i0 -c4 /lustre/lustre/d0.dbench-c08 striped dir -i3 -c4 /lustre/l==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=59637 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: Exit config file opensfs.sh c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c02: Exit config file opensfs.sh c01: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 17:09:39 (1438042179) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 17:12:43 (1438042363) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 9 times, and counting... sleeping 1583 seconds... riped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 stririped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 strir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dir -i2 -c4 /lustre/lustre/d0.dd-c03 striped dirriped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 strir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir -i1 -c4 /lustre/lustre/d0.dd-c06 striped dir==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=61428 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: Exit config file opensfs.sh c05: Exit config file opensfs.sh c04: Exit config file opensfs.sh c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds02: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 17:39:39 (1438043979) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 17:42:37 (1438044157) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 9 times, and counting... sleeping 1587 seconds... ped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 stripeped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 stripeped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 stripe==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=63224 DURATION=86400 PERIOD=1800 Wait mds1 recovery complete before doing next failover... affected facets: mds1 mds01: Exit config file opensfs.sh mds01: *.lustre-MDT0000.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c03: Exit config file opensfs.sh c03: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: Exit config file opensfs.sh c07: Exit config file opensfs.sh c06: Exit config file opensfs.sh c02: Exit config file opensfs.sh c04: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds2 mds3 mds4 are in FULL state before doing next failover... mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds1 Failing mds1 on mds01 + runas -u di.wang ssh 192.168.0.1 pm -0 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds01] Command completed successfully reboot facets: mds1 + runas -u di.wang ssh 192.168.0.1 pm -1 mds01 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds01] Command completed successfully Failover mds1 to mds01 18:09:39 (1438045779) waiting for mds01 network 900 secs ... waiting ping -c 1 -w 3 mds01, 895 secs left ... waiting ping -c 1 -w 3 mds01, 890 secs left ... waiting ping -c 1 -w 3 mds01, 885 secs left ... waiting ping -c 1 -w 3 mds01, 880 secs left ... waiting ping -c 1 -w 3 mds01, 875 secs left ... waiting ping -c 1 -w 3 mds01, 870 secs left ... waiting ping -c 1 -w 3 mds01, 865 secs left ... waiting ping -c 1 -w 3 mds01, 860 secs left ... waiting ping -c 1 -w 3 mds01, 855 secs left ... waiting ping -c 1 -w 3 mds01, 850 secs left ... waiting ping -c 1 -w 3 mds01, 845 secs left ... waiting ping -c 1 -w 3 mds01, 840 secs left ... waiting ping -c 1 -w 3 mds01, 835 secs left ... waiting ping -c 1 -w 3 mds01, 830 secs left ... waiting ping -c 1 -w 3 mds01, 825 secs left ... waiting ping -c 1 -w 3 mds01, 820 secs left ... waiting ping -c 1 -w 3 mds01, 815 secs left ... waiting ping -c 1 -w 3 mds01, 810 secs left ... waiting ping -c 1 -w 3 mds01, 805 secs left ... waiting ping -c 1 -w 3 mds01, 800 secs left ... waiting ping -c 1 -w 3 mds01, 795 secs left ... waiting ping -c 1 -w 3 mds01, 790 secs left ... waiting ping -c 1 -w 3 mds01, 785 secs left ... 18:12:43 (1438045963) network interface is UP mount facets: mds1 Starting mds1: -o user_xattr,acl /dev/sdf1 /lustre/mds1 mds01: mount.lustre: increased /sys/block/sdf/queue/max_sectors_kb from 1024 to 16383 mds01: Exit config file opensfs.sh Started lustre-MDT0000 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds1 has failed over 10 times, and counting... sleeping 1583 seconds... d dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i0 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i3 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i1 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped dir -i2 -c4 /lustre/lustre/d0.tar-c01 striped d dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i1 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i3 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i2 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped dir -i0 -c4 /lustre/lustre/d0.tar-c04 striped d dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i1 -c4 /lustre/lustre/d0.tar-c07 striped dir -i0 -c4 /lustre/lustre/d0.tar-c07 striped dir -i2 -c4 /lustre/lustre/d0.tar-c07 striped dir -i3 -c4 /lustre/lustre/d0.tar-c07 striped ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=65028 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c08: Exit config file opensfs.sh c01: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds01: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 18:39:38 (1438047578) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 18:42:34 (1438047754) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 10 times, and counting... sleeping 1582 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=66829 DURATION=86400 PERIOD=1800 Wait mds2 recovery complete before doing next failover... affected facets: mds2 mds02: Exit config file opensfs.sh mds02: *.lustre-MDT0001.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c06: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c03: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c02: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds3 mds4 are in FULL state before doing next failover... mds03: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh Starting failover on mds2 Failing mds2 on mds02 + runas -u di.wang ssh 192.168.0.1 pm -0 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds02] Command completed successfully reboot facets: mds2 + runas -u di.wang ssh 192.168.0.1 pm -1 mds02 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds02] Command completed successfully Failover mds2 to mds02 19:10:00 (1438049400) waiting for mds02 network 900 secs ... waiting ping -c 1 -w 3 mds02, 895 secs left ... waiting ping -c 1 -w 3 mds02, 890 secs left ... waiting ping -c 1 -w 3 mds02, 885 secs left ... waiting ping -c 1 -w 3 mds02, 880 secs left ... waiting ping -c 1 -w 3 mds02, 875 secs left ... waiting ping -c 1 -w 3 mds02, 870 secs left ... waiting ping -c 1 -w 3 mds02, 865 secs left ... waiting ping -c 1 -w 3 mds02, 860 secs left ... waiting ping -c 1 -w 3 mds02, 855 secs left ... waiting ping -c 1 -w 3 mds02, 850 secs left ... waiting ping -c 1 -w 3 mds02, 845 secs left ... waiting ping -c 1 -w 3 mds02, 840 secs left ... waiting ping -c 1 -w 3 mds02, 835 secs left ... waiting ping -c 1 -w 3 mds02, 830 secs left ... waiting ping -c 1 -w 3 mds02, 825 secs left ... waiting ping -c 1 -w 3 mds02, 820 secs left ... waiting ping -c 1 -w 3 mds02, 815 secs left ... waiting ping -c 1 -w 3 mds02, 810 secs left ... waiting ping -c 1 -w 3 mds02, 805 secs left ... waiting ping -c 1 -w 3 mds02, 800 secs left ... waiting ping -c 1 -w 3 mds02, 795 secs left ... waiting ping -c 1 -w 3 mds02, 790 secs left ... 19:12:57 (1438049577) network interface is UP mount facets: mds2 Starting mds2: -o user_xattr,acl /dev/sdb1 /lustre/mds2 mds02: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds02: Exit config file opensfs.sh Started lustre-MDT0001 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds2 has failed over 10 times, and counting... sleeping 1516 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=68699 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c06: Exit config file opensfs.sh c03: Exit config file opensfs.sh c02: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c08: Exit config file opensfs.sh c07: Exit config file opensfs.sh c04: Exit config file opensfs.sh c06: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c02: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds4 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh mds03: Exit config file opensfs.sh Starting failover on mds3 Failing mds3 on mds03 + runas -u di.wang ssh 192.168.0.1 pm -0 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds03] Command completed successfully reboot facets: mds3 + runas -u di.wang ssh 192.168.0.1 pm -1 mds03 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds03] Command completed successfully Failover mds3 to mds03 19:39:43 (1438051183) waiting for mds03 network 900 secs ... waiting ping -c 1 -w 3 mds03, 895 secs left ... waiting ping -c 1 -w 3 mds03, 890 secs left ... waiting ping -c 1 -w 3 mds03, 885 secs left ... waiting ping -c 1 -w 3 mds03, 880 secs left ... waiting ping -c 1 -w 3 mds03, 875 secs left ... waiting ping -c 1 -w 3 mds03, 870 secs left ... waiting ping -c 1 -w 3 mds03, 865 secs left ... waiting ping -c 1 -w 3 mds03, 860 secs left ... waiting ping -c 1 -w 3 mds03, 855 secs left ... waiting ping -c 1 -w 3 mds03, 850 secs left ... waiting ping -c 1 -w 3 mds03, 845 secs left ... waiting ping -c 1 -w 3 mds03, 840 secs left ... waiting ping -c 1 -w 3 mds03, 835 secs left ... waiting ping -c 1 -w 3 mds03, 830 secs left ... waiting ping -c 1 -w 3 mds03, 825 secs left ... waiting ping -c 1 -w 3 mds03, 820 secs left ... waiting ping -c 1 -w 3 mds03, 815 secs left ... waiting ping -c 1 -w 3 mds03, 810 secs left ... waiting ping -c 1 -w 3 mds03, 805 secs left ... waiting ping -c 1 -w 3 mds03, 800 secs left ... waiting ping -c 1 -w 3 mds03, 795 secs left ... waiting ping -c 1 -w 3 mds03, 790 secs left ... 19:42:41 (1438051361) network interface is UP mount facets: mds3 Starting mds3: -o user_xattr,acl /dev/sda1 /lustre/mds3 mds03: mount.lustre: increased /sys/block/sda/queue/max_sectors_kb from 1024 to 16383 mds03: Exit config file opensfs.sh Started lustre-MDT0002 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds3 has failed over 10 times, and counting... sleeping 1529 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=70486 DURATION=86400 PERIOD=1800 Wait mds4 recovery complete before doing next failover... affected facets: mds4 mds04: Exit config file opensfs.sh mds04: *.lustre-MDT0003.recovery_status status: COMPLETE Checking clients are in FULL state before doing next failover... c02: Exit config file opensfs.sh c03: Exit config file opensfs.sh c06: Exit config file opensfs.sh c08: Exit config file opensfs.sh c04: Exit config file opensfs.sh c07: Exit config file opensfs.sh c05: Exit config file opensfs.sh c01: Exit config file opensfs.sh c02: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c03: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c06: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c08: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c04: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c07: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c05: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec c01: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec Checking mds1 mds2 mds3 are in FULL state before doing next failover... mds02: Exit config file opensfs.sh mds03: Exit config file opensfs.sh mds01: Exit config file opensfs.sh mds04: Exit config file opensfs.sh Starting failover on mds4 Failing mds4 on mds04 + runas -u di.wang ssh 192.168.0.1 pm -0 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-0] [mds04] Command completed successfully reboot facets: mds4 + runas -u di.wang ssh 192.168.0.1 pm -1 mds04 running as uid/gid/euid/egid 503/503/503/503, groups: [ssh] [192.168.0.1] [pm] [-1] [mds04] Command completed successfully Failover mds4 to mds04 20:09:44 (1438052984) waiting for mds04 network 900 secs ... waiting ping -c 1 -w 3 mds04, 895 secs left ... waiting ping -c 1 -w 3 mds04, 890 secs left ... waiting ping -c 1 -w 3 mds04, 885 secs left ... waiting ping -c 1 -w 3 mds04, 880 secs left ... waiting ping -c 1 -w 3 mds04, 875 secs left ... waiting ping -c 1 -w 3 mds04, 870 secs left ... waiting ping -c 1 -w 3 mds04, 865 secs left ... waiting ping -c 1 -w 3 mds04, 860 secs left ... waiting ping -c 1 -w 3 mds04, 855 secs left ... waiting ping -c 1 -w 3 mds04, 850 secs left ... waiting ping -c 1 -w 3 mds04, 845 secs left ... waiting ping -c 1 -w 3 mds04, 840 secs left ... waiting ping -c 1 -w 3 mds04, 835 secs left ... waiting ping -c 1 -w 3 mds04, 830 secs left ... waiting ping -c 1 -w 3 mds04, 825 secs left ... waiting ping -c 1 -w 3 mds04, 820 secs left ... waiting ping -c 1 -w 3 mds04, 815 secs left ... waiting ping -c 1 -w 3 mds04, 810 secs left ... waiting ping -c 1 -w 3 mds04, 805 secs left ... waiting ping -c 1 -w 3 mds04, 800 secs left ... waiting ping -c 1 -w 3 mds04, 795 secs left ... waiting ping -c 1 -w 3 mds04, 790 secs left ... 20:12:42 (1438053162) network interface is UP mount facets: mds4 Starting mds4: -o user_xattr,acl /dev/sdb1 /lustre/mds4 mds04: mount.lustre: increased /sys/block/sdb/queue/max_sectors_kb from 1024 to 16383 mds04: Exit config file opensfs.sh Started lustre-MDT0003 panic_on_lbug=0 ==== Checking the clients loads AFTER failover -- failure NOT OK mds4 has failed over 11 times, and counting... sleeping 1532 seconds... ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=72283 DURATION=86400 PERIOD=1800 Wait mds3 recovery complete before doing next failover... affected facets: mds3 mds03: Exit config file opensfs.sh mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1470 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1465 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1460 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1455 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1450 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1445 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1440 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1435 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1430 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1425 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1420 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1415 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1410 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1405 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1400 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1395 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1390 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1385 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1380 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1375 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1370 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1365 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1360 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1355 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1350 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1345 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1340 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1335 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1330 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1325 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1320 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1315 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1310 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1305 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1300 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1295 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1290 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1285 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1280 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1275 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1270 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1265 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1260 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1255 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1250 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1245 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1240 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1235 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1230 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1225 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1220 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1215 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1210 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1205 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1200 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1195 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1190 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1185 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1180 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1175 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1170 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1165 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1160 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1155 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1150 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1145 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1140 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1135 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1130 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1125 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1120 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1115 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1110 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1105 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1100 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1095 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1090 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1085 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1080 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1075 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1070 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1065 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1060 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1055 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1050 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1045 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1040 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1035 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1030 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1025 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1020 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1015 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1010 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1005 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 1000 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 995 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 990 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 985 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 980 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 975 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 970 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 965 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 960 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 955 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 950 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 945 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 940 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 935 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 930 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 925 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 920 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 915 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 910 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 905 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 900 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 895 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 890 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 885 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 880 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 875 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 870 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 865 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 860 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 855 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 850 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 845 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 840 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 835 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 830 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 825 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 820 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 815 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 810 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 805 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 800 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 795 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 790 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 785 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 780 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 775 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 770 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 765 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 760 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 755 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 750 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 745 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 740 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 735 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 730 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 725 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 720 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 715 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 710 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 705 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 700 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 695 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 690 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 685 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 680 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 675 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 670 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 665 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 660 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 655 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 650 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 645 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 640 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 635 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 630 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 625 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 620 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 615 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 610 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 605 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 600 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 595 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 590 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 585 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 580 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 575 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 570 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 565 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 560 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 555 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 550 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 545 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 540 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 535 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 530 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 525 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 520 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 515 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 510 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 505 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 500 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 495 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 490 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 485 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 480 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 475 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 470 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 465 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 460 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 455 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 450 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 445 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 440 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 435 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 430 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 425 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 420 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 415 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 410 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 405 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 400 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 395 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 390 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 385 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 380 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 375 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 370 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 365 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 360 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 355 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 350 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 345 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 340 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 335 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 330 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 325 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 320 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 315 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 310 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 305 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 300 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 295 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 290 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 285 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 280 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 275 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 270 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 265 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 260 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 255 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 250 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 245 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 240 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 235 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 230 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 225 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 220 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 215 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 210 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 205 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 200 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 195 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 190 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 185 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 180 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 175 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 170 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 165 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 160 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 155 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 150 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 145 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 140 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 135 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 130 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 125 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 120 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 115 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 110 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 105 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 100 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 95 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 90 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 85 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 80 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 75 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 70 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 65 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 60 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 55 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 50 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 45 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 40 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 35 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 30 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 25 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 20 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 15 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 10 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 5 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status status: RECOVERING mds03: Waiting 0 secs for *.lustre-MDT0002.recovery_status recovery done. status: RECOVERING mds03: *.lustre-MDT0002.recovery_status recovery not done in 1475 sec. status: RECOVERING mds3 recovery is not completed! 2015-07-27 21:04:06 Terminating clients loads ... Duration: 86400 Server failover period: 1800 seconds Exited after: 72283 seconds Number of failovers before exit: mds1: 10 times mds2: 10 times mds3: 10 times mds4: 11 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times Status: FAIL: rc=7 /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 15853 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16000 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16147 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16294 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16441 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16588 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16735 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 103: 16882 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS MDSCOUNT=$MDSCOUNT NODENUM=$nodenum run_${load}.sh" Dumping lctl log to /tmp/test_logs/2015-07-27/000218/recovery-mds-scale.test_failover_mds.*.1438056246.log mds01: open /proc/sys/lnet/dump_kernel failed: No such file or directory mds01: open(dump_kernel) failed: No such file or directory mds01: Host key verification failed. mds01: rsync: connection unexpectedly closed (0 bytes received so far) [sender] mds01: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6] oss01: Host key verification failed. oss01: rsync: connection unexpectedly closed (0 bytes received so far) [sender] oss01: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6] mds04: Host key verification failed. mds04: rsync: connection unexpectedly closed (0 bytes received so far) [sender] mds04: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6] mds03: Host key verification failed. mds03: rsync: connection unexpectedly closed (0 bytes received so far) [sender] mds03: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6] mds02: Host key verification failed. mds02: rsync: connection unexpectedly closed (0 bytes received so far) [sender] mds02: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6] oss02: Host key verification failed. oss02: rsync: connection unexpectedly closed (0 bytes received so far) [sender] oss02: rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]