[LU-10421] mds-survey test 1: Timeout occurred after 426 mins, last suite running was mds-survey, restarting cluster to continue tests

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.3
• Labels: dne zfs
• Environment: onyx, full DNE; servers: el7.4, zfs, branch master, v2.10.56, b3678; clients: el7.4, branch master, v2.10.56, b3678
• Severity: 3
• Assignee: John Hammond
• Reporter: James Casper

Description

session: https://testing.hpdd.intel.com/test_sessions/9e3f4edc-daff-4e9c-bb2c-5e501afcb7bf
test set: https://testing.hpdd.intel.com/test_sets/ba56cb40-e0c8-11e7-9840-52540065bddc

From MDS console:

[22053.144258] LustreError: 13506:0:(echo_client.c:1795:echo_md_lookup()) lookup MDT0001-tests: rc = -2
[22053.145264] LustreError: 13506:0:(echo_client.c:2027:echo_md_destroy_internal()) Can't find child MDT0001-tests: rc = -2
[22053.781142] LustreError: 13611:0:(echo_client.c:1795:echo_md_lookup()) lookup MDT0001-tests3: rc = -2
[22053.782164] LustreError: 13611:0:(echo_client.c:1795:echo_md_lookup()) Skipped 2 previous similar messages
[22053.783133] LustreError: 13611:0:(echo_client.c:2027:echo_md_destroy_internal()) Can't find child MDT0001-tests3: rc = -2
[22053.784222] LustreError: 13611:0:(echo_client.c:2027:echo_md_destroy_internal()) Skipped 2 previous similar messages
[22055.866749] LustreError: 13891:0:(echo_client.c:1795:echo_md_lookup()) lookup MDT0003-tests: rc = -2
[22055.867931] LustreError: 13891:0:(echo_client.c:2027:echo_md_destroy_internal()) Can't find child MDT0003-tests: rc = -2

[22177.268865] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008

[22177.270372] IP: [<ffffffffc0bdb913>] lu_object_alloc+0x73/0x310 [obdclass]
[22177.271432] PGD 48733067 PUD 3cfc0067 PMD 0 
[22177.272157] Oops: 0002 [#1] SMP 
[22177.272692] Modules linked in: obdecho(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfsd pcspkr i2c_piix4 joydev virtio_balloon parport_pc i2c_core parport nfs_acl lockd auth_rpcgss grace sunrpc ip_tables ata_generic pata_acpi ext4 mbcache jbd2 ata_piix libata virtio_blk 8139too crct10dif_pclmul crct10dif_common floppy crc32c_intel virtio_pci virtio_ring serio_raw virtio 8139cp mii
[22177.287656] CPU: 1 PID: 19364 Comm: lctl Tainted: P           OE  ------------   3.10.0-693.5.2.el7_lustre.x86_64 #1
[22177.289215] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[22177.290055] task: ffff88003ef69fa0 ti: ffff880049e5c000 task.ti: ffff880049e5c000
[22177.291188] RIP: 0010:[<ffffffffc0bdb913>]  [<ffffffffc0bdb913>] lu_object_alloc+0x73/0x310 [obdclass]
[22177.292617] RSP: 0018:ffff880049e5fb20  EFLAGS: 00010246
[22177.293373] RAX: 00000002400090a0 RBX: ffff8800528d0e40 RCX: 0000000000000000
[22177.294437] RDX: 0000000000000007 RSI: 0000000000000000 RDI: ffff88004885a000
[22177.295492] RBP: ffff880049e5fb68 R08: 0000000000000000 R09: ffff88004885a000
[22177.296546] R10: 000000000000000d R11: 0000000000000fff R12: ffff88004885a000
[22177.297624] R13: ffff880049e5fc08 R14: ffff88005a97a1f8 R15: 0000000000000000
[22177.298634] FS:  00007f0dfe043740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[22177.299832] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[22177.300688] CR2: 0000000000000008 CR3: 000000001c64d000 CR4: 00000000000406e0
[22177.301787] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[22177.302862] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[22177.303911] Stack:
[22177.304231]  ffff880053233000 ffffffffc0bd91f3 0000000000000000 ffff880058793e18
[22177.305487]  ffff8800528d0e40 0000000000000000 ffff880049e5fc08 ffff88005a97a1f8
[22177.306635]  ffff880053233000 ffff880049e5fbd0 ffffffffc0bdbd7c ffff880058793e18
[22177.307918] Call Trace:
[22177.308341]  [<ffffffffc0bd91f3>] ? htable_lookup+0x153/0x170 [obdclass]
[22177.309359]  [<ffffffffc0bdbd7c>] lu_object_find_at+0x16c/0x290 [obdclass]
[22177.310377]  [<ffffffffc11bfa9e>] echo_md_dir_stripe_choose.isra.43+0x26e/0x680 [obdecho]
[22177.311601]  [<ffffffffc05d77eb>] ? cfs_hash_spin_unlock+0xb/0x10 [libcfs]
[22177.312625]  [<ffffffffc11c0d6c>] echo_md_handler.isra.45+0xebc/0x2c20 [obdecho]
[22177.313708]  [<ffffffffc11c6891>] echo_client_iocontrol+0x1091/0x1ba0 [obdecho]
[22177.314799]  [<ffffffffc0bbc459>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[22177.315936]  [<ffffffffc0ba714d>] class_handle_ioctl+0x18cd/0x1dd0 [obdclass]
[22177.316937]  [<ffffffff811b1e81>] ? handle_mm_fault+0x691/0xfa0
[22177.317792]  [<ffffffff812b1a98>] ? security_capable+0x18/0x20
[22177.318674]  [<ffffffffc0b8c602>] obd_class_ioctl+0xd2/0x170 [obdclass]
[22177.319675]  [<ffffffff812151bd>] do_vfs_ioctl+0x33d/0x540
[22177.320472]  [<ffffffff816b0456>] ? trace_do_page_fault+0x56/0x150
[22177.321376]  [<ffffffff81215461>] SyS_ioctl+0xa1/0xc0
[22177.322137]  [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b

Issue Links

• duplicates: LU-6249 mds-survey test_1: test failed to respond and timed out
• is duplicated by: LU-10636 mds-survey test_1: timeout
• is duplicated by: LU-10758 mds-survey: timeout