[LU-10062] parallel-scale-nfsv3 test_compilebench: compilebench failed: 1

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.7, Lustre 2.10.8
• Severity: 3
• Assignee: WC Triage
• Reporter: James Casper

Description

https://testing.whamcloud.com/test_sets/4bf0cda6-9d0a-11e7-b778-5254006e85c2

From test_log:

 
create dir kernel-0 222MB in 490.97 seconds (0.45 MB/s)
Traceback (most recent call last):
  File "./compilebench", line 576, in 
    mbs = run_directory(dset.unpatched, dirname, "create dir")
  File "./compilebench", line 245, in run_directory
    fp.close()
IOError: [Errno 5] Input/output error
 parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:5289:error()
  = /usr/lib64/lustre/tests/functions.sh:335:run_compilebench()
  = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:98:test_compilebench()
  = /usr/lib64/lustre/tests/test-framework.sh:5565:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5604:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:5451:run_test()
  = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:100:main()

From the MDS console:

 
============================================== 21:33:54 \(1505795634\)
05:07:40:[  384.742296] Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test compilebench: compilebench ============================================== 21:33:54 (1505795634)
05:07:40:[  384.933753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench.13751 -i 2         -r 2 --makej
05:07:40:[  385.094371] Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench.13751 -i 2 -r 2 --makej
05:07:40:[ 2404.251434] LustreError: 11-0: lustre-OST0001-osc-ffff8800001e4800: operation ost_write to node 10.9.4.18@tcp failed: rc = -107
05:07:40:[ 2404.257382] Lustre: lustre-OST0001-osc-ffff8800001e4800: Connection to lustre-OST0001 (at 10.9.4.18@tcp) was lost; in progress operations using this service will wait for recovery to complete
05:07:40:[ 2406.281680] LustreError: 167-0: lustre-OST0001-osc-ffff8800001e4800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
05:07:40:[ 2406.287600] Lustre: 3553:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xcfc2:0x0]/ may get corrupted (rc -108)
05:07:40:[ 2406.288316] Lustre: 3552:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xc488:0x0]/ may get corrupted (rc -108)
...
05:17:02:[ 2406.293786] Lustre: 3552:0:(llite_lib.c:2624:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.9.4.19@tcp:/lustre/fid: [0x200000401:0xd039:0x0]/ may get corrupted (rc -108)
05:17:02:[ 2407.172942] ------------[ cut here ]------------
05:17:02:[ 2407.175861] WARNING: CPU: 0 PID: 8522 at fs/nfsd/nfsproc.c:804 nfserrno+0x58/0x70 [nfsd]
05:17:02:[ 2407.178986] nfsd: non-standard errno: -108
05:17:02:[ 2407.179955] Lustre: lustre-OST0001-osc-ffff8800001e4800: Connection restored to 10.9.4.18@tcp (at 10.9.4.18@tcp)
05:17:02:[ 2407.185134] Modules linked in: osc(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon joydev pcspkr nfsd i2c_piix4 parport_pc parport nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm 8139too drm virtio_blk ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata 8139cp serio_raw i2c_core virtio_pci mii virtio_ring virtio floppy
05:17:02:[ 2407.213461] CPU: 0 PID: 8522 Comm: nfsd Tainted: G           OE  ------------   3.10.0-693.1.1.el7_lustre.x86_64 #1
05:17:02:[ 2407.216760] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
05:17:02:[ 2407.219714]  ffff880060cffc88 000000001c745b52 ffff880060cffc38 ffffffff816a3d6d
05:17:02:[ 2407.222799]  ffff880060cffc78 ffffffff810879c8 00000324810c4765 ffff880063039a80
05:17:02:[ 2407.225884]  ffff8800179fb024 ffff880063f0a008 ffff8800179fb000 000000000000001c
05:17:02:[ 2407.228939] Call Trace:
05:17:02:[ 2407.231514]  [<ffffffff816a3d6d>] dump_stack+0x19/0x1b
05:17:02:[ 2407.234317]  [<ffffffff810879c8>] __warn+0xd8/0x100
05:17:02:[ 2407.236957]  [<ffffffff81087a4f>] warn_slowpath_fmt+0x5f/0x80
05:17:02:[ 2407.239697]  [<ffffffffc0347608>] nfserrno+0x58/0x70 [nfsd]
05:17:02:[ 2407.242329]  [<ffffffffc03553ad>] encode_post_op_attr.isra.3+0x6d/0xe0 [nfsd]
05:17:02:[ 2407.245120]  [<ffffffffc034d53c>] ? nfsd_write+0x11c/0x290 [nfsd]
05:17:02:[ 2407.247743]  [<ffffffffc03559d2>] encode_wcc_data.isra.5+0x72/0xb0 [nfsd]
05:17:02:[ 2407.250577]  [<ffffffffc0356b7f>] nfs3svc_encode_writeres+0x4f/0xc0 [nfsd]
05:17:02:[ 2407.253277]  [<ffffffffc0345613>] nfsd_dispatch+0x153/0x280 [nfsd]
05:17:02:[ 2407.256059]  [<ffffffffc025a453>] svc_process_common+0x453/0x6f0 [sunrpc]
05:17:02:[ 2407.258672]  [<ffffffffc025a7f3>] svc_process+0x103/0x190 [sunrpc]
05:17:02:[ 2407.261414]  [<ffffffffc0344eff>] nfsd+0xdf/0x150 [nfsd]
05:17:02:[ 2407.263818]  [<ffffffffc0344e20>] ? nfsd_destroy+0x80/0x80 [nfsd]
05:17:02:[ 2407.266434]  [<ffffffff810b098f>] kthread+0xcf/0xe0
05:17:02:[ 2407.268730]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
05:17:02:[ 2407.271269]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
05:17:02:[ 2407.273529]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
05:17:02:[ 2407.276029] ---[ end trace 92f46a85c11f036c ]---
05:17:02:[ 2407.464242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1 
05:17:02:[ 2407.625853] Lustre: DEBUG MARKER: parallel-scale-nfsv3 test_compilebench: @@@@@@ FAIL: compilebench failed: 1

Issue Links

• is related to: LU-14343 parallel-scale-nfsv4 test compilebench fails with 'compilebench failed: 1'
• is related to: LU-2661 Failure on test suite parallel-scale-nfsv3 test_compilebench: compilebench failed: 1