[LU-2968] OST Crash on test suite sanity, subtest test_65k

Fri, 15 Mar 2013 17:36:27 +0000
This issue was created by maloo for Nathaniel Clark <nathaniel.l.clark@intel.com>
This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/d3d3d4d8-8c4e-11e2-aa89-52540035b04c.
The sub-test test_65k failed with the following error:
test failed to respond and timed out
Info required for matching: sanity 65k
OST Crash
07:19:51:Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs == 07:19:41 (1363184381)
07:19:51:Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 10.10.17.28@tcp) reconnecting
07:19:51:Lustre: lustre-OST0000: deleting orphan objects from 0x0:832 to 864
07:19:51:Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 10.10.17.28@tcp) reconnecting
07:19:51:Lustre: Skipped 6 previous similar messages
07:19:51:Lustre: lustre-OST0000: deleting orphan objects from 0x0:832 to 896
07:19:51:Lustre: Skipped 6 previous similar messages
07:19:51:LustreError: 10669:0:(ldlm_resource.c:1171:ldlm_resource_get()) lvbo_init failed for resource 864: rc -2
07:19:51:------------[ cut here ]------------
07:19:51:WARNING: at lib/list_debug.c:51 list_del+0x8d/0xa0() (Tainted: P           ---------------   )
07:19:51:Hardware name: KVM
07:19:51:list_del corruption. next->prev should be ffff8800703f7d00, but was 5a5a5a5a5a5a5a5a
07:19:51:Modules linked in: osp(U) ofd(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) nfsd exportfs autofs4 nfs lockd fscache nfs_acl auth_rpcgss sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
07:19:51:Pid: 10669, comm: ll_ost00_014 Tainted: P           ---------------    2.6.32-279.19.1.el6_lustre.gc4681d8.x86_64 #1
07:19:51:Call Trace:
07:19:51: <IRQ>  [<ffffffff8106a1e7>] ? warn_slowpath_common+0x87/0xc0
07:19:51: [<ffffffff8106a2d6>] ? warn_slowpath_fmt+0x46/0x50
07:19:51: [<ffffffff81279efd>] ? list_del+0x8d/0xa0
07:19:51: [<ffffffffa006a236>] ? blk_done+0x46/0x110 [virtio_blk]
07:19:51: [<ffffffff810de334>] ? __rcu_process_callbacks+0x54/0x330
07:19:51: [<ffffffffa005b1dc>] ? vring_interrupt+0x3c/0xd0 [virtio_ring]
07:19:51: [<ffffffff810d8b40>] ? handle_IRQ_event+0x60/0x170
07:19:51: [<ffffffff810729ef>] ? __do_softirq+0x11f/0x1e0
07:19:51: [<ffffffff810db20e>] ? handle_edge_irq+0xde/0x180
07:19:51: [<ffffffff8100de89>] ? handle_irq+0x49/0xa0
07:19:51: [<ffffffff814f217c>] ? do_IRQ+0x6c/0xf0
07:19:51: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
07:19:51: <EOI>  [<ffffffff81275335>] ? memset+0x45/0xc0
07:19:51: [<ffffffffa0e6475f>] ? ofd_lvbo_free+0xbf/0xe0 [ofd]
07:19:51: [<ffffffffa08899f8>] ? ldlm_resource_putref+0x128/0x280 [ptlrpc]
07:19:51: [<ffffffffa088bbb6>] ? ldlm_resource_get+0x5e6/0x750 [ptlrpc]
07:19:51: [<ffffffffa0885915>] ? ldlm_lock_create+0x55/0xa30 [ptlrpc]
07:19:51: [<ffffffffa08a381e>] ? ldlm_cli_enqueue_local+0xbe/0x5d0 [ptlrpc]
07:19:51: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:51: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:51: [<ffffffffa0e4a8a0>] ? ofd_destroy_by_fid+0x160/0x380 [ofd]
07:19:51: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:51: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:51: [<ffffffffa0e4da09>] ? ofd_create+0xdb9/0x1470 [ofd]
07:19:51: [<ffffffffa0e2247c>] ? ost_handle+0x356c/0x46f0 [ost]
07:19:51: [<ffffffffa05ca154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
07:19:51: [<ffffffffa08dd29c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
07:19:51: [<ffffffffa05be5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
07:19:51: [<ffffffffa08d48d9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
07:19:51: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
07:19:51: [<ffffffffa08de7e5>] ? ptlrpc_main+0xb75/0x1870 [ptlrpc]
07:19:51: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:51: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
07:19:51: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:51: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:52: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
07:19:52:---[ end trace 80b36efad3167dcc ]---
07:19:52:BUG: scheduling while atomic: ll_ost00_014/10669/0x10010000
07:19:52:Modules linked in: osp(U) ofd(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) nfsd exportfs autofs4 nfs lockd fscache nfs_acl auth_rpcgss sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
07:19:52:CPU 0 
07:19:52:Modules linked in: osp(U) ofd(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) nfsd exportfs autofs4 nfs lockd fscache nfs_acl auth_rpcgss sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
07:19:52:
07:19:52:Pid: 10669, comm: ll_ost00_014 Tainted: P        W  ---------------    2.6.32-279.19.1.el6_lustre.gc4681d8.x86_64 #1 Red Hat KVM
07:19:52:RIP: 0010:[<ffffffff81275335>]  [<ffffffff81275335>] memset+0x45/0xc0
07:19:52:RSP: 0018:ffff88005b47f8f8  EFLAGS: 00000203
07:19:52:RAX: 5a5a5a5a5a5a5a5a RBX: ffff88005b47f910 RCX: 0000000003e52c31
07:19:52:RDX: 0000000000000000 RSI: 000000000000005a RDI: ffff880067b13800
07:19:52:RBP: ffffffff8100b9ce R08: 00000000ffffff0a R09: 0000000000000000
07:19:52:R10: ffff880060fc44c0 R11: fffffffffffffffe R12: ffff88005b47f880
07:19:52:R13: 0000000000000002 R14: 0000000000000000 R15: 0000049300000000
07:19:52:FS:  00007f5744b2f700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
07:19:52:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
07:19:52:CR2: 00007fbc7955a340 CR3: 0000000077ee7000 CR4: 00000000000006f0
07:19:52:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
07:19:52:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
07:19:52:Process ll_ost00_014 (pid: 10669, threadinfo ffff88005b47e000, task ffff88005b452080)
07:19:52:Stack:
07:19:52: ffffffffa0e6475f ffff880064c76440 ffff880064c76440 ffff88005b47f950
07:19:52:<d> ffffffffa08899f8 ffffc90005b24000 ffff88000000000d 0000000000000360
07:19:52:<d> 00000000fffffffe ffff88005dc9c8e0 0000000000000000 ffff88005b47f9c0
07:19:52:Call Trace:
07:19:52: [<ffffffffa0e6475f>] ? ofd_lvbo_free+0xbf/0xe0 [ofd]
07:19:52: [<ffffffffa08899f8>] ? ldlm_resource_putref+0x128/0x280 [ptlrpc]
07:19:52: [<ffffffffa088bbb6>] ? ldlm_resource_get+0x5e6/0x750 [ptlrpc]
07:19:52: [<ffffffffa0885915>] ? ldlm_lock_create+0x55/0xa30 [ptlrpc]
07:19:52: [<ffffffffa08a381e>] ? ldlm_cli_enqueue_local+0xbe/0x5d0 [ptlrpc]
07:19:52: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:52: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:52: [<ffffffffa0e4a8a0>] ? ofd_destroy_by_fid+0x160/0x380 [ofd]
07:19:52: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:52: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:52: [<ffffffffa0e4da09>] ? ofd_create+0xdb9/0x1470 [ofd]
07:19:52: [<ffffffffa0e2247c>] ? ost_handle+0x356c/0x46f0 [ost]
07:19:52: [<ffffffffa05ca154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
07:19:52: [<ffffffffa08dd29c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
07:19:52: [<ffffffffa05be5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
07:19:52: [<ffffffffa08d48d9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
07:19:52: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
07:19:53: [<ffffffffa08de7e5>] ? ptlrpc_main+0xb75/0x1870 [ptlrpc]
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:53: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:53: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
07:19:53:Code: 41 83 e1 07 75 7e 44 89 d9 c1 e9 06 74 38 0f 1f 84 00 00 00 00 00 ff c9 48 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 <48> 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 66 0f 1f 
07:19:53:Call Trace:
07:19:53: [<ffffffffa0e6475f>] ? ofd_lvbo_free+0xbf/0xe0 [ofd]
07:19:53: [<ffffffffa08899f8>] ? ldlm_resource_putref+0x128/0x280 [ptlrpc]
07:19:53: [<ffffffffa088bbb6>] ? ldlm_resource_get+0x5e6/0x750 [ptlrpc]
07:19:53: [<ffffffffa0885915>] ? ldlm_lock_create+0x55/0xa30 [ptlrpc]
07:19:53: [<ffffffffa08a381e>] ? ldlm_cli_enqueue_local+0xbe/0x5d0 [ptlrpc]
07:19:53: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:53: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:53: [<ffffffffa0e4a8a0>] ? ofd_destroy_by_fid+0x160/0x380 [ofd]
07:19:53: [<ffffffffa08a26d0>] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
07:19:53: [<ffffffffa08a3d30>] ? ldlm_completion_ast+0x0/0x960 [ptlrpc]
07:19:53: [<ffffffffa0e4da09>] ? ofd_create+0xdb9/0x1470 [ofd]
07:19:53: [<ffffffffa0e2247c>] ? ost_handle+0x356c/0x46f0 [ost]
07:19:53: [<ffffffffa05ca154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
07:19:53: [<ffffffffa08dd29c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
07:19:53: [<ffffffffa05be5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
07:19:53: [<ffffffffa08d48d9>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
07:19:53: [<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
07:19:53: [<ffffffffa08de7e5>] ? ptlrpc_main+0xb75/0x1870 [ptlrpc]
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:53: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:53: [<ffffffffa08ddc70>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
07:19:54: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Whamcloud Community JIRA

[LU-2968] OST Crash on test suite sanity, subtest test_65k
