Details
-
Bug
-
Resolution: Duplicate
-
Minor
-
None
-
Lustre 2.11.0
-
None
-
Clients/Servers - 2.10.57
-
3
-
9223372036854775807
Description
replay-dual test_26 - Timeout occurred after 854 mins, last suite running was replay-dual, restarting cluster to continue tests
^^^^^^^^^^^^^ DO NOT REMOVE LINE ABOVE ^^^^^^^^^^^^^
This issue was created by maloo for Saurabh Tandan <saurabh.tandan@intel.com>
This issue relates to the following test suite run:
https://testing.hpdd.intel.com/test_sets/d9b9f5c8-fd3f-11e7-a6ad-52540065bddc
test_26 failed with the following error:
Timeout occurred after 854 mins, last suite running was replay-dual, restarting cluster to continue tests
test_log:
onyx-30vm4: onyx-30vm4.onyx.hpdd.intel.com: executing set_default_debug -1 all 4
CMD: onyx-30vm4 e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: onyx-30vm4 e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
CMD: onyx-30vm4 e2label /dev/lvm-Role_MDS/P1 2>/dev/null
Started lustre-MDT0000
CMD: onyx-30vm1.onyx.hpdd.intel.com,onyx-30vm2 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
onyx-30vm2: onyx-30vm2.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
onyx-30vm1: onyx-30vm1.onyx.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
onyx-30vm1: CMD: onyx-30vm1.onyx.hpdd.intel.com lctl get_param -n at_max
onyx-30vm2: CMD: onyx-30vm2.onyx.hpdd.intel.com lctl get_param -n at_max
onyx-30vm2: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 177 sec
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/firstboot': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/drbd': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/sudo': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/hypervkvp': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/roundup': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/vhostmd': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/dnssec': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/chrome': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/sge': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/brctl': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/cyphesis': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/games': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/clock': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/modules/100/milter': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/policy.kern': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/active/commit_num': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/semanage.read.LOCK': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/semanage.trans.LOCK': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/.policy.sha512': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/d26.replay-dual/run_tar/etc/selinux/targeted/contexts': Cannot send after transport endpoint shutdown rm: fts_read failed: Cannot send after transport endpoint shutdown onyx-30vm1: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 183 sec /usr/lib64/lustre/tests/replay-dual.sh: line 982: kill: (5954) - No such process replay-dual test_26: @@@@@@ FAIL: tar 5954 missing Trace dump: