[LU-8544] recovery-double-scale test_pairwise_fail: start client on trevis-54vm5 failed

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.9.0
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/72a0fd32-6033-11e6-aa74-5254006e85c2.

The sub-test test_pairwise_fail failed with the following error:

start client on trevis-54vm5 failed

test logs:

CMD: trevis-54vm5 test -f /tmp/client-load.pid &&
        { kill -s TERM \$(cat /tmp/client-load.pid); rm -f /tmp/client-load.pid; }
+ pm -h powerman --off trevis-54vm5
Command completed successfully
+ pm -h powerman --on trevis-54vm5
Command completed successfully
14:43:16 (1470926596) waiting for trevis-54vm5 network 900 secs ...
waiting ping -c 1 -w 3 trevis-54vm5, 895 secs left ...
waiting ping -c 1 -w 3 trevis-54vm5, 890 secs left ...
waiting ping -c 1 -w 3 trevis-54vm5, 885 secs left ...
waiting ping -c 1 -w 3 trevis-54vm5, 880 secs left ...
14:43:48 (1470926628) network interface is UP
CMD: trevis-54vm5 hostname
pdsh@trevis-54vm1: trevis-54vm5: mcmd: connect failed: Connection refused
CMD: trevis-54vm5 hostname
Reintegrating trevis-54vm5
Starting client: trevis-54vm5:  -o user_xattr,flock trevis-54vm7:trevis-54vm3:/lustre /mnt/lustre
CMD: trevis-54vm5 mkdir -p /mnt/lustre
CMD: trevis-54vm5 mount -t lustre -o user_xattr,flock trevis-54vm7:trevis-54vm3:/lustre /mnt/lustre
CMD: trevis-54vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck\" \"all -lnet -lnd -pinger\" 4 
trevis-54vm5: stat: cannot read file system information for ‘/mnt/lustre’: Input/output error
 recovery-double-scale test_pairwise_fail: @@@@@@ FAIL: start client on trevis-54vm5 failed
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4804:error()
  = /usr/lib64/lustre/tests/recovery-double-scale.sh:72:reboot_recover_node()
  = /usr/lib64/lustre/tests/recovery-double-scale.sh:160:failover_pair()
  = /usr/lib64/lustre/tests/recovery-double-scale.sh:251:test_pairwise_fail()
  = /usr/lib64/lustre/tests/test-framework.sh:5068:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5107:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4954:run_test()
  = /usr/lib64/lustre/tests/recovery-double-scale.sh:303:main()

Issue Links

• is related to: LU-6670 Hard Failover recovery-small test_28: post-failover df: 1
• is related to: LU-4039 Failure on test suite replay-single test_90: wrong stripe: f0, uuid: lustre-OST0000_UUID
• is related to: LU-8526 replay-single test_90: @@@@@@ FAIL: wrong stripe: all, uuid: lustre-OST0000_UUID
• is related to: LU-8731 lfs df exits with status 0 on failures
• is related to: LU-6610 lfs df -h query hangs when OST1 is unmounted/offline manually
• is related to: LU-5788 recovery-double-scale test_pairwise_fail: no enough free disk space
• is related to: LU-6655 MDS LBUG: (ldlm_lib.c:2277:target_queue_recovery_request()) ASSERTION( req->rq_export->exp_lock_replay_needed ) failed
• is related to: LU-7759 umount hanging in modern distros when OST is unavailable