Lustre: DEBUG MARKER: /usr/sbin/lctl mark == recovery-mds-scale test failover_ost: failover OST ================================================ 13:35:20 \(1345840520\)
Lustre: DEBUG MARKER: == recovery-mds-scale test failover_ost: failover OST ================================================ 13:35:20 (1345840520)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: dd on client-26vm1
Lustre: DEBUG MARKER: Started client load: dd on client-26vm1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: tar on client-26vm2
Lustre: DEBUG MARKER: Started client load: tar on client-26vm2
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: dbench on client-26vm5
Lustre: DEBUG MARKER: Started client load: dbench on client-26vm5
Lustre: DEBUG MARKER: /usr/sbin/lctl mark ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=900
Lustre: DEBUG MARKER: ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=900
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait ost6 recovery complete before doing next failover...
Lustre: DEBUG MARKER: Wait ost6 recovery complete before doing next failover...
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Checking clients are in FULL state before doing next failover...
Lustre: DEBUG MARKER: Checking clients are in FULL state before doing next failover...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[^M]*.ost_server_uuid in FULL state after 0 sec
Lustre: lustre-OST0005: Export ffff8800720f2400 already connecting from 10.10.4.150@tcp
Lustre: Skipped 26 previous similar messages
Lustre: lustre-OST0005: Client 33159da3-4758-1c50-a4ed-0926567964fa (at 10.10.4.152@tcp) reconnecting
Lustre: Skipped 355 previous similar messages
Lustre: lustre-OST0005: Client 33159da3-4758-1c50-a4ed-0926567964fa (at 10.10.4.152@tcp) refused reconnection, still busy with 6 active RPCs
Lustre: Skipped 355 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl mark rpc : @@@@@@ FAIL: can\'t put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: rpc : @@@@@@ FAIL: can't put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests
Lustre: DEBUG MARKER: /usr/sbin/lctl mark rpc : @@@@@@ FAIL: can\'t put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: /usr/sbin/lctl mark rpc : @@@@@@ FAIL: can\'t put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: rpc : @@@@@@ FAIL: can't put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /tmp/test_logs/1345840602/rpc..debug_log.$(hostname -s).1345841358.log;
dmesg > /tmp/test_logs/1345840602/rpc..dmesg.$(hostname -s).1345841358.log
Lustre: DEBUG MARKER: rpc : @@@@@@ FAIL: can't put import for osc.lustre-OST0005-osc-[^M]*.ost_server_uuid into FULL state after 662 sec, have DISCONN
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:./../utils:/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
cannot allocate a tage (2)
Lustre: lustre-OST0005: Export ffff8800720f2400 already connecting from 10.10.4.150@tcp
Lustre: Skipped 37 previous similar messages
Lustre: DEBUG MARKER: rsync -az /tmp/test_logs/1345840602/rpc..*.1345841358.log client-26vm5.lab.whamcloud.com:/tmp/test_logs/1345840602
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /tmp/test_logs/1345840602/rpc..debug_log.$(hostname -s).1345841372.log;
dmesg > /tmp/test_logs/1345840602/rpc..dmesg.$(hostname -s).1345841372.log
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /tmp/test_logs/1345840604/rpc..debug_log.$(hostname -s).1345841379.log;
dmesg > /tmp/test_logs/1345840604/rpc..dmesg.$(hostname -s).1345841379.log
Lustre: DEBUG MARKER: rsync -az /tmp/test_logs/1345840602/rpc..*.1345841372.log client-26vm1.lab.whamcloud.com:/tmp/test_logs/1345840602
Lustre: DEBUG MARKER: rsync -az /tmp/test_logs/1345840604/rpc..*.1345841379.log client-26vm2.lab.whamcloud.com:/tmp/test_logs/1345840604
Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-mds-scale test_failover_ost: @@@@@@ FAIL: import is not in FULL state
Lustre: DEBUG MARKER: recovery-mds-scale test_failover_ost: @@@@@@ FAIL: import is not in FULL state