Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Lustre 2.11.0, Lustre 2.10.7, Lustre 2.10.8
-
None
-
3
-
9223372036854775807
Description
https://testing.whamcloud.com/test_sets/4bf0cda6-9d0a-11e7-b778-5254006e85c2
From test_log:
create dir kernel-0 222MB in 490.97 seconds (0.45 MB/s)
Traceback (most recent call last):
File "./compilebench", line 576, in
mbs = run_directory(dset.unpatched, dirname, "create dir")
File "./compilebench", line 245, in run_directory
fp.close()
IOError: [Errno 5] Input/output error
parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:5289:error()
= /usr/lib64/lustre/tests/functions.sh:335:run_compilebench()
= /usr/lib64/lustre/tests/parallel-scale-nfs.sh:98:test_compilebench()
= /usr/lib64/lustre/tests/test-framework.sh:5565:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:5604:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:5451:run_test()
= /usr/lib64/lustre/tests/parallel-scale-nfs.sh:100:main()
From the MDS console:
============================================== 21:33:54 \(1505795634\) 05:07:40:[ 384.742296] Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test compilebench: compilebench ============================================== 21:33:54 (1505795634) 05:07:40:[ 384.933753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench.13751 -i 2 -r 2 --makej 05:07:40:[ 385.094371] Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench.13751 -i 2 -r 2 --makej 05:07:40:[ 2404.251434] LustreError: 11-0: lustre-OST0001-osc-ffff8800001e4800: operation ost_write to node 10.9.4.18@tcp failed: rc = -107 05:07:40:[ 2404.257382] Lustre: lustre-OST0001-osc-ffff8800001e4800: Connection to lustre-OST0001 (at 10.9.4.18@tcp) was lost; in progress operations using this service will wait for recovery to complete 05:07:40:[ 2406.281680] LustreError: 167-0: lustre-OST0001-osc-ffff8800001e4800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. 05:07:40:[ 2406.287600] Lustre: 3553:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xcfc2:0x0]/ may get corrupted (rc -108) 05:07:40:[ 2406.288316] Lustre: 3552:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xc488:0x0]/ may get corrupted (rc -108) ... 05:17:02:[ 2406.293786] Lustre: 3552:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xd039:0x0]/ may get corrupted (rc -108) 05:17:02:[ 2407.172942] ------------[ cut here ]------------ 05:17:02:[ 2407.175861] WARNING: CPU: 0 PID: 8522 at fs/nfsd/nfsproc.c:804 nfserrno+0x58/0x70 [nfsd] 05:17:02:[ 2407.178986] nfsd: non-standard errno: -108 05:17:02:[ 2407.179955] Lustre: lustre-OST0001-osc-ffff8800001e4800: Connection restored to 10.9.4.18@tcp (at 10.9.4.18@tcp) 05:17:02:[ 2407.185134] Modules linked in: osc(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon joydev pcspkr nfsd i2c_piix4 parport_pc parport nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm 8139too drm virtio_blk ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata 8139cp serio_raw i2c_core virtio_pci mii virtio_ring virtio floppy 05:17:02:[ 2407.213461] CPU: 0 PID: 8522 Comm: nfsd Tainted: G OE ------------ 3.10.0-693.1.1.el7_lustre.x86_64 #1 05:17:02:[ 2407.216760] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 05:17:02:[ 2407.219714] ffff880060cffc88 000000001c745b52 ffff880060cffc38 ffffffff816a3d6d 05:17:02:[ 2407.222799] ffff880060cffc78 ffffffff810879c8 00000324810c4765 ffff880063039a80 05:17:02:[ 2407.225884] ffff8800179fb024 ffff880063f0a008 ffff8800179fb000 000000000000001c 05:17:02:[ 2407.228939] Call Trace: 05:17:02:[ 2407.231514] [<ffffffff816a3d6d>] dump_stack+0x19/0x1b 05:17:02:[ 2407.234317] [<ffffffff810879c8>] __warn+0xd8/0x100 05:17:02:[ 2407.236957] [<ffffffff81087a4f>] warn_slowpath_fmt+0x5f/0x80 05:17:02:[ 2407.239697] [<ffffffffc0347608>] nfserrno+0x58/0x70 [nfsd] 05:17:02:[ 2407.242329] [<ffffffffc03553ad>] encode_post_op_attr.isra.3+0x6d/0xe0 [nfsd] 05:17:02:[ 2407.245120] [<ffffffffc034d53c>] ? nfsd_write+0x11c/0x290 [nfsd] 05:17:02:[ 2407.247743] [<ffffffffc03559d2>] encode_wcc_data.isra.5+0x72/0xb0 [nfsd] 05:17:02:[ 2407.250577] [<ffffffffc0356b7f>] nfs3svc_encode_writeres+0x4f/0xc0 [nfsd] 05:17:02:[ 2407.253277] [<ffffffffc0345613>] nfsd_dispatch+0x153/0x280 [nfsd] 05:17:02:[ 2407.256059] [<ffffffffc025a453>] svc_process_common+0x453/0x6f0 [sunrpc] 05:17:02:[ 2407.258672] [<ffffffffc025a7f3>] svc_process+0x103/0x190 [sunrpc] 05:17:02:[ 2407.261414] [<ffffffffc0344eff>] nfsd+0xdf/0x150 [nfsd] 05:17:02:[ 2407.263818] [<ffffffffc0344e20>] ? nfsd_destroy+0x80/0x80 [nfsd] 05:17:02:[ 2407.266434] [<ffffffff810b098f>] kthread+0xcf/0xe0 05:17:02:[ 2407.268730] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40 05:17:02:[ 2407.271269] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90 05:17:02:[ 2407.273529] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40 05:17:02:[ 2407.276029] ---[ end trace 92f46a85c11f036c ]--- 05:17:02:[ 2407.464242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1 05:17:02:[ 2407.625853] Lustre: DEBUG MARKER: parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1