[LU-14625] recovery-small test_138: Timeout occurred after 721 mins, last suite running was recovery-small

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Artem Blagodarenko <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/cc96a8db-f56d-4d78-8649-5acac8b61e7a

test_138 failed with the following error:

Timeout occurred after 721 mins, last suite running was recovery-small

CMD: trevis-64vm8 /usr/sbin/lctl get_param -n health_check
CMD: trevis-64vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config bash rpc.sh set_default_debug \"-1\" \"all\" 4
trevis-64vm8: CMD: trevis-64vm8 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm8 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm7 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm8.trevis.whamcloud.com /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: trevis-64vm8.trevis.whamcloud.com: executing set_default_debug -1 all 4
CMD: trevis-64vm8 zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]

{3}[0-9]{4}'
pdsh@trevis-64vm5: trevis-64vm8: ssh exited with exit code 1
CMD: trevis-64vm8 zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null
Started lustre-MDT0000
CMD: trevis-64vm8 grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Stopping /mnt/lustre-mds1 (opts on trevis-64vm8
CMD: trevis-64vm8 umount -d /mnt/lustre-mds1
CMD: trevis-64vm8 lsmod | grep lnet > /dev/null &&
lctl dl | grep ' ST ' || true
CMD: trevis-64vm8 ! zpool list -H lustre-mdt1 >/dev/null 2>&1 ||
grep -q ^lustre-mdt1/ /proc/mounts ||
zpool export lustre-mdt1
CMD: trevis-64vm8 /usr/sbin/lctl set_param fail_loc=0
fail_loc=0
CMD: trevis-64vm8 mkdir -p /mnt/lustre-mds1
CMD: trevis-64vm8 lsmod | grep zfs >&/dev/null || modprobe zfs;
zpool list -H lustre-mdt1 >/dev/null 2>&1 ||
zpool import -f -o cachefile=none -o failmode=panic -d /dev/lvm-Role_MDS lustre-mdt1
CMD: trevis-64vm8 zfs get -H -o value lustre:svname lustre-mdt1/mdt1
Starting mds1: lustre-mdt1/mdt1 /mnt/lustre-mds1
CMD: trevis-64vm8 mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1
CMD: trevis-64vm8 /usr/sbin/lctl get_param -n health_check
CMD: trevis-64vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/sbin:/sbin:/bin::/sbin:/bin:/usr/sbin: NAME=autotest_config bash rpc.sh set_default_debug \"-1\" \"all\" 4
trevis-64vm8: CMD: trevis-64vm8 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm8 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm7 /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: CMD: trevis-64vm8.trevis.whamcloud.com /usr/sbin/lctl get_param -n version 2>/dev/null
trevis-64vm8: trevis-64vm8.trevis.whamcloud.com: executing set_default_debug -1 all 4
CMD: trevis-64vm8 zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}

[0-9]

{4}

'
pdsh@trevis-64vm5: trevis-64vm8: ssh exited with exit code 1
CMD: trevis-64vm8 zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null
Started lustre-MDT0000
Starting client trevis-64vm5.trevis.whamcloud.com,trevis-64vm6: -o user_xattr,flock trevis-64vm8@tcp:/lustre /mnt/lustre
CMD: trevis-64vm5.trevis.whamcloud.com,trevis-64vm6 mkdir -p /mnt/lustre
CMD: trevis-64vm5.trevis.whamcloud.com,trevis-64vm6
running=\$(mount | grep -c /mnt/lustre' ');
rc=0;
if [ \$running -eq 0 ] ; then
mkdir -p /mnt/lustre;
mount -t lustre -o user_xattr,flock trevis-64vm8@tcp:/lustre /mnt/lustre;
rc=\$?;
fi;
exit \$rc

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
recovery-small test_138 - Timeout occurred after 721 mins, last suite running was recovery-small