[LU-5347] replay-ost-single test_3: tee: standard output: Input/output error

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.2
• Environment: Lustre Build: https://build.hpdd.intel.com/job/lustre-master/2048/; Distro/Arch: RHEL6.5/x86_64
• Severity: 3
• Assignee: Jian Yu
• Reporter: Jian Yu

Description

replay-ost-single test 3 failed as follows:

== replay-ost-single test 3: Fail OST during write, with verification == 20:09:46 (1400123386)
Failing ost1 on shadow-49vm8
CMD: shadow-49vm8 grep -c /mnt/ost1' ' /proc/mounts
Stopping /mnt/ost1 (opts:) on shadow-49vm8
CMD: shadow-49vm8 umount -d /mnt/ost1
CMD: shadow-49vm8 lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '
reboot facets: ost1
Failover ost1 to shadow-49vm8
20:10:04 (1400123404) waiting for shadow-49vm8 network 900 secs ...
20:10:04 (1400123404) network interface is UP
CMD: shadow-49vm8 hostname
mount facets: ost1
CMD: shadow-49vm8 test -b /dev/lvm-Role_OSS/P1
Starting ost1:   /dev/lvm-Role_OSS/P1 /mnt/ost1
CMD: shadow-49vm8 mkdir -p /mnt/ost1; mount -t lustre   		                   /dev/lvm-Role_OSS/P1 /mnt/ost1
CMD: shadow-49vm8 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/bin:/bin:/sbin:/usr/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super\" \"all -lnet -lnd -pinger\" 4 
CMD: shadow-49vm8 e2label /dev/lvm-Role_OSS/P1 2>/dev/null
Started lustre-OST0000
CMD: shadow-49vm1,shadow-49vm2.shadow.whamcloud.com PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/bin:/bin:/sbin:/usr/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid 
shadow-49vm1: CMD: shadow-49vm1.shadow.whamcloud.com lctl get_param -n at_max
shadow-49vm2: CMD: shadow-49vm2.shadow.whamcloud.com lctl get_param -n at_max
shadow-49vm1: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 95 sec
tee: standard output: Input/output error
shadow-49vm2: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 105 sec
1280+0 records in
1280+0 records out
5242880 bytes (5.2 MB) copied, 125.153 s, 41.9 kB/s
tee: write error
 replay-ost-single test_3: @@@@@@ FAIL: test_3 failed with 1 

Debug log on client:

00000080:00200000:0.0:1400123511.721003:0:28098:0:(file.c:1229:ll_file_io_generic()) iot: 1, result: -5
00000100:00100000:1.0:1400123511.726097:0:20485:0:(client.c:1851:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_0:c890eb56-964e-2a0e-1f85-b5461794e3dc:0:1468135597178740:10.1.6.62@tcp:4
00000008:00080000:1.0:1400123511.726121:0:20486:0:(osc_request.c:1769:brw_interpret()) lustre-OST0000-osc-ffff88006b95d000: resend cross eviction for object: 0x0:15338, rc = -5.
00000100:00100000:1.0:1400123511.726335:0:20486:0:(client.c:1851:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:c890eb56-964e-2a0e-1f85-b5461794e3dc:0:1468135597149296:10.1.6.62@tcp:4
00000008:00080000:1.0:1400123511.726341:0:20486:0:(osc_request.c:1769:brw_interpret()) lustre-OST0000-osc-ffff88006b95d000: resend cross eviction for object: 0x0:15338, rc = -5.

Maloo report: http://testing.hpdd.intel.com/test_sets/1786067c-dc5a-11e3-8b61-52540035b04c