[LU-6860] mds-survey test_2: MDS crashed

Related — Mon, 20 Jul 2015 16:09:20 +0000

This issue was created by maloo for sarah_lw <wei3.liu@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/502b1446-2776-11e5-8996-5254006e85c2.

The sub-test test_2 failed with the following error:

test failed to respond and timed out

MDS console

23:29:46:Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 == 23:29:38 (1436484578)
23:29:46:Lustre: DEBUG MARKER: lctl dl
23:29:46:Lustre: Echo OBD driver; http://www.lustre.org/
23:29:46:LustreError: 11974:0:(echo_client.c:1676:echo_md_lookup()) lookup MDT0000-tests: rc = -2
23:29:46:LustreError: 11974:0:(echo_client.c:1875:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2
23:29:46:LustreError: 12040:0:(echo_client.c:1676:echo_md_lookup()) lookup MDT0000-tests3: rc = -2
23:29:46:LustreError: 12040:0:(echo_client.c:1676:echo_md_lookup()) Skipped 2 previous similar messages
23:29:46:LustreError: 12040:0:(echo_client.c:1875:echo_md_destroy_internal()) Can't find child MDT0000-tests3: rc = -2
23:29:46:LustreError: 12040:0:(echo_client.c:1875:echo_md_destroy_internal()) Skipped 2 previous similar messages
23:29:46:BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
23:29:46:IP: [<ffffffff8105bff2>] task_rq_lock+0x42/0xa0
23:29:46:PGD 6eec1067 PUD 6eec3067 PMD 0 
23:29:46:Oops: 0000 [#1] SMP 
23:29:46:last sysfs file: /sys/devices/system/cpu/online
23:29:46:CPU 0 
23:29:46:Modules linked in: obdecho(U) osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgs(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: obdecho]
23:29:46:
23:29:46:Pid: 12152, comm: lctl Tainted: P           ---------------    2.6.32-504.23.4.el6_lustre.x86_64 #1 Red Hat KVM
23:29:46:RIP: 0010:[<ffffffff8105bff2>]  [<ffffffff8105bff2>] task_rq_lock+0x42/0xa0
23:29:46:RSP: 0018:ffff880071ac9888  EFLAGS: 00010086
23:29:46:RAX: 0000000000000286 RBX: 0000000000015900 RCX: ffff880079aa47f8
23:29:46:RDX: 0000000000000286 RSI: ffff880071ac98e0 RDI: 0000000000000000
23:29:46:RBP: ffff880071ac98a8 R08: 0000000000000002 R09: 5a5a5a5a5a5a5a5a
23:29:46:R10: 5a5a5a5a5a5a5a5a R11: 5a5a5a5a5a5a5a5a R12: 0000000000000000
23:29:46:R13: ffff880071ac98e0 R14: 0000000000000000 R15: 000000000000000f
23:29:46:FS:  00007f9b41973700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
23:29:46:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
23:29:46:CR2: 0000000000000008 CR3: 000000006f4c4000 CR4: 00000000000006f0
23:29:46:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
23:29:46:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
23:29:46:Process lctl (pid: 12152, threadinfo ffff880071ac8000, task ffff88006f926ab0)
23:29:46:Stack:
23:29:46: 0000000000000000 ffff88004d327ca0 0000000000000000 0000000000000000
23:29:46:<d> ffff880071ac9918 ffffffff8106483c ffff880071ac98d8 ffffffffa0559b38
23:29:46:<d> ffff880078157338 ffff880071949e78 ffff880071ac9958 0000000000000286
23:29:46:Call Trace:
23:29:46: [<ffffffff8106483c>] try_to_wake_up+0x3c/0x3e0
23:29:46: [<ffffffffa0559b38>] ? echo_object_free+0x278/0x5e0 [obdecho]
23:29:46: [<ffffffff81064c35>] wake_up_process+0x15/0x20
23:29:57: [<ffffffff8152b6e2>] __mutex_unlock_slowpath+0x42/0x60
23:29:57: [<ffffffff8152b65b>] mutex_unlock+0x1b/0x20
23:29:57: [<ffffffffa086ab81>] lu_site_purge+0x401/0x4f0 [obdclass]
23:29:57: [<ffffffffa086b0b1>] lu_object_limit+0x71/0x80 [obdclass]
23:29:57: [<ffffffffa086b1f0>] lu_object_find_try+0x130/0x260 [obdclass]
23:29:57: [<ffffffffa086b3d1>] lu_object_find_at+0xb1/0xe0 [obdclass]
23:29:57: [<ffffffffa129e406>] ? mdd_lookup+0xe6/0x180 [mdd]
23:29:57: [<ffffffffa055c3b3>] echo_md_create_internal+0x153/0x640 [obdecho]
23:29:57: [<ffffffffa056656d>] echo_md_handler+0x176d/0x1e30 [obdecho]
23:29:57: [<ffffffffa0568658>] echo_client_iocontrol+0x1a28/0x2c50 [obdecho]
23:29:57: [<ffffffffa082f21c>] class_handle_ioctl+0x16dc/0x2260 [obdclass]
23:29:57: [<ffffffffa08132ab>] obd_class_ioctl+0x4b/0x190 [obdclass]
23:29:57: [<ffffffff811a3e12>] vfs_ioctl+0x22/0xa0
23:29:57: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
23:29:57: [<ffffffff811a3fb4>] do_vfs_ioctl+0x84/0x580
23:29:57: [<ffffffff81040f2c>] ? kvm_clock_read+0x1c/0x20
23:29:57: [<ffffffff81040f39>] ? kvm_clock_get_cycles+0x9/0x10
23:29:57: [<ffffffff810a9c47>] ? getnstimeofday+0x57/0xe0
23:29:57: [<ffffffff811a4531>] sys_ioctl+0x81/0xa0
23:29:57: [<ffffffff810e608e>] ? __audit_syscall_exit+0x25e/0x290
23:29:57: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
23:29:57:Code: 89 74 24 18 0f 1f 44 00 00 48 c7 c3 00 59 01 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 89 55 00 <49> 8b 44 24 08 49 89 de 8b 40 18 4c 03 34 c5 a0 0b c0 81 4c 89 
23:29:57:RIP  [<ffffffff8105bff2>] task_rq_lock+0x42/0xa0
23:29:57: RSP <ffff880071ac9888>
23:29:57:CR2: 0000000000000008
23:29:57:Initializing cgroup subsys cpuset

Whamcloud Community JIRA

[LU-6860] mds-survey test_2: MDS crashed