[LU-15243] recovery-double-scale test_pairwise fails because of "pm -h powerman --on" failed: onyx-32-virsh: login timeout

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Elena <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/2c3d8111-965f-4fb7-9a4b-ad27b99311fd

test_pairwise_fail failed with the following error:

Restart of ost1 failed!

the reason of failure is failed "power down" command completed with errors:
+ pm -h powerman --off onyx-32vm2
onyx-32-virsh: login timeout
Command completed with errors
waiting ! ping -w 3 -c 1 onyx-32vm2, 4 secs left ...
waiting ! ping -w 3 -c 1 onyx-32vm2, 3 secs left ...
waiting ! ping -w 3 -c 1 onyx-32vm2, 2 secs left ...
waiting ! ping -w 3 -c 1 onyx-32vm2, 1 secs left ...
waiting for onyx-32vm2 to fail attempts=3
onyx-32vm2 still pingable after power down! attempts=3
reboot facets: ost1,ost2,ost3,ost4,ost5,ost6,ost7
+ pm -h powerman --on onyx-32vm2
onyx-32-virsh: login timeout
Command completed with errors
CMD: onyx-32vm2 hostname
Failover ost1 to onyx-32vm1
Failover ost2 to onyx-32vm1
Failover ost3 to onyx-32vm1
Failover ost4 to onyx-32vm1
Failover ost5 to onyx-32vm1
Failover ost6 to onyx-32vm1
Failover ost7 to onyx-32vm1
CMD: onyx-32vm1 hostname
mount facets: ost1,ost2,ost3,ost4,ost5,ost6,ost7
CMD: onyx-32vm1 dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
pdsh@onyx-78vm4: onyx-32vm1: ssh exited with exit code 1
CMD: onyx-32vm1 test -b /dev/lvm-Role_OSS/P1
CMD: onyx-32vm1 blockdev --getsz /dev/lvm-Role_OSS/P1 2>/dev/null
CMD: onyx-32vm1 dmsetup create ost1_flakey --table \"0 22953984 linear /dev/lvm-Role_OSS/P1 0\"
CMD: onyx-32vm1 dmsetup mknodes >/dev/null 2>&1
CMD: onyx-32vm1 test -b /dev/mapper/ost1_flakey
CMD: onyx-32vm1 e2label /dev/mapper/ost1_flakey
Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1
CMD: onyx-32vm1 mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1
onyx-32vm1: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Invalid argument
onyx-32vm1: This may have multiple causes.
onyx-32vm1: Are the mount options correct?
onyx-32vm1: Check the syslog for more info.
pdsh@onyx-78vm4: onyx-32vm1: ssh exited with exit code 22
Start of /dev/mapper/ost1_flakey on ost1 failed 22
recovery-double-scale test_pairwise_fail: @@@@@@ FAIL: Restart of ost1 failed!

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
recovery-double-scale test_pairwise_fail - Restart of ost1 failed!

Issue Links

• is related to: LU-15349 recovery-double-scale: failed because of "pm -h powerman --on <client>" failed