[LU-1499] parallel-scale-nfsv4, subtest test_iorfpp

• Type: Bug
• Resolution: Incomplete
• Priority: Blocker
• Affects Version/s: Lustre 2.3.0, Lustre 2.1.2, Lustre 2.1.4, Lustre 2.5.1, Lustre 2.7.0
• Labels: mn1
• Environment: Lustre Tag: v2_1_2_RC3; Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/89/; Distro/Arch: RHEL5.8/x86_64
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for yujian <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/edb73380-b329-11e1-a6d7-52540035b04c.

The sub-test test_iorfpp failed with the following error:

** error **
ERROR in aiori-POSIX.c (line 256): transfer failed.
ERROR: Input/output error
** exiting **

parallel-scale-nfsv4 test_iorfpp: @@@@@@ FAIL: ior failed! 1

Info required for matching: parallel-scale-nfsv4 iorfpp

Console log on MDS showed that:

10:30:59:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test iorfpp: iorfpp ========================================================== 10:30:56 (1339349456)
10:31:19:LustreError: 11-0: an error occurred while communicating with 10.10.4.167@tcp. The ost_write operation failed with -107
10:31:19:LustreError: Skipped 2 previous similar messages
10:31:19:Lustre: lustre-OST0000-osc-ffff81006c110800: Connection to lustre-OST0000 (at 10.10.4.167@tcp) was lost; in progress operations using this service will wait for recovery to complete
10:31:19:Lustre: Skipped 2 previous similar messages
10:31:19:LustreError: 11-0: an error occurred while communicating with 10.10.4.167@tcp. The obd_ping operation failed with -107
10:31:20:LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
10:31:21:LustreError: 3215:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff81006ae4dc00 x1404345569553508/t0(0) o4->lustre-OST0000-osc-ffff81006c110800@10.10.4.167@tcp:6/4 lens 456/416 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
10:31:22:LustreError: 3215:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810042141800 x1404345569553507/t0(0) o4->lustre-OST0000-osc-ffff81006c110800@10.10.4.167@tcp:6/4 lens 472/416 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
10:31:22:LustreError: 3215:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff810016181000 x1404345569553531/t0(0) o4->lustre-OST0000-osc-ffff81006c110800@10.10.4.167@tcp:6/4 lens 472/416 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
10:31:22:LustreError: 3215:0:(client.c:1060:ptlrpc_import_delay_req()) Skipped 22 previous similar messages
10:31:27:LustreError: 3217:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff81005e151400 x1404345569553575/t0(0) o101->lustre-OST0000-osc-ffff81006c110800@10.10.4.167@tcp:28/4 lens 296/352 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
10:31:27:LustreError: 3217:0:(client.c:1060:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
10:34:05:Lustre: DEBUG MARKER: parallel-scale-nfsv4 test_iorfpp: @@@@@@ FAIL: ior failed! 1

Console log on OSS showed that:

10:30:59:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test iorfpp: iorfpp ========================================================== 10:30:56 (1339349456)
10:31:16:Lustre: 8348:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1339349469/real 1339349469]  req@ffff81002bb25400 x1404345556224852/t0(0) o105->lustre-OST0000@10.10.4.166@tcp:15/16 lens 344/192 e 0 to 1 dl 1339349476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
10:31:17:LustreError: 138-a: lustre-OST0000: A client on nid 10.10.4.166@tcp was evicted due to a lock completion callback time out: rc -107
10:31:21:Lustre: 8361:0:(ldlm_lib.c:933:target_handle_connect()) lustre-OST0000: connection from d2bf4c4f-6776-3492-f34a-d2b1bfc5d2b3@10.10.4.166@tcp t1460255 exp 0000000000000000 cur 1339349480 last 0
10:31:21:Lustre: 8361:0:(ldlm_lib.c:933:target_handle_connect()) Skipped 2 previous similar messages
10:34:04:Lustre: DEBUG MARKER: parallel-scale-nfsv4 test_iorfpp: @@@@@@ FAIL: ior failed! 1

Issue Links

• Trackbacks: Lustre 2.1.2 release testing tracker Lustre 2.1.2 RC3 Tag: v212RC3 Build: