[LU-7166] Failover: recovery-small test_61: test failed to respond and timed out

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Affects Version/s: Lustre 2.8.0
• Environment: Client: lustre-master# 3166, SLES11 SP3 ; Server: lustre-master# 3166, RHEL 7
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/45ebfbec-52de-11e5-b61c-5254006e85c2.

The sub-test test_61 failed with the following error:

test failed to respond and timed out

mds dmesg:

[  324.583610] Lustre: DEBUG MARKER: == recovery-small test 61: Verify to not reuse orphan objects - bug 17025 ============================ 01:01:28 (1441267288)
[  324.768561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version
[  325.119814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version
[  325.444742] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc             > /dev/null 2>&1
[  325.761165] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT*.connect_flags
[  325.930762] LustreError: 1704:0:(ldlm_lockd.c:677:ldlm_handle_ast_error()) ### client (nid 10.1.4.193@tcp) returned error from blocking AST (req status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff880069a87a00/0x280320e8c568af9f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 10.1.4.193@tcp remote: 0x7314c03cb6a7c1ad expref: 7 pid: 1705 timeout: 4295093226 lvb_type: 0
[  325.934334] LustreError: 138-a: lustre-MDT0000: A client on nid 10.1.4.193@tcp was evicted due to a lock blocking callback time out: rc -107
[  326.110147] Lustre: DEBUG MARKER: sync; sync; sync
[  326.931245] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno
[  327.239483] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly
[  327.387196] LustreError: 3188:0:(osd_handler.c:1380:osd_ro()) *** setting lustre-MDT0000 read-only ***
[  327.388887] Turning device dm-0 (0xfc00000) read-only
[  327.548071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000
[  327.703668] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[  328.291521] Lustre: DEBUG MARKER: grep -c /mnt/mds1' ' /proc/mounts
[  328.578880] Lustre: DEBUG MARKER: umount -d /mnt/mds1
[  328.750566] Lustre: Failing over lustre-MDT0000
[  331.905101] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.1.4.191@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[  331.908769] LustreError: Skipped 5 previous similar messages
[  333.253260] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.1.4.187@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[  334.827156] Lustre: 3384:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1441267293/real 1441267293]  req@ffff8800001ded00 x1511278203896684/t0(0) o251->MGC10.1.4.188@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1441267299 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[  334.836234] Removing read-only on unknown block (0xfc00000)
[  334.870635] Lustre: server umount lustre-MDT0000 complete
[  335.031375] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '