[LU-1909] performance-sanity subtest test_3: Oops: RIP: __do_softirq+0x73/0x1e0

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Affects Version/s: Lustre 2.3.0
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Maloo

Description

This issue was created by maloo for yujian <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/ffd15c18-fc84-11e1-a4a6-52540035b04c.

The sub-test test_3 failed with the following error:

test failed to respond and timed out

Info required for matching: performance-sanity 3

Lustre Build: http://build.whamcloud.com/job/lustre-b2_3/17

Console log on MDS (client-26vm7):

11:19:16:Lustre: DEBUG MARKER: ===== mdsrate-create-small.sh
11:19:16:__end_that: dev ?: type=ffff8800, flags=ffff8800
11:19:16:BUG: scheduling while atomic: mdt00_000/20284/0x10010100
11:19:16:Modules linked in: osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) lustre(U) obdfilter(U) ost(U) cmm(U) mdt(U) mdd(U) mds(U) mgs(U) obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic jbd2 nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
11:19:16:CPU 0 
11:19:16:Modules linked in: osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) lustre(U) obdfilter(U) ost(U) cmm(U) mdt(U) mdd(U) mds(U) mgs(U) obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic jbd2 nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
11:19:16:
11:19:16:Pid: 20284, comm: mdt00_000 Not tainted 2.6.32-279.5.1.el6_lustre.g634f764.x86_64 #1 Red Hat KVM
11:19:16:RIP: 0010:[<ffffffff81073e73>]  [<ffffffff81073e73>] __do_softirq+0x73/0x1e0
11:19:16:RSP: 0018:ffff880002203ee0  EFLAGS: 00000206
11:19:16:RAX: 0000000000011400 RBX: ffff880002203f40 RCX: 0000000000000020
11:19:16:RDX: ffff88005ab4dfd8 RSI: 000000000000dd43 RDI: 0000000000000380
11:19:16:RBP: ffffffff8100ba53 R08: ffff88000220e0e0 R09: 00001dbb48795100
11:19:16:R10: 0000000000000010 R11: 0000000000000007 R12: ffff880002203e60
11:19:17:R13: ffff880026d28000 R14: ffff880026d28028 R15: ffff88005ab4dfd8
11:19:17:FS:  00007fca0b14f700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
11:19:17:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
11:19:17:CR2: 00007f440da9b936 CR3: 00000000375cb000 CR4: 00000000000006f0
11:19:17:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
11:19:17:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
11:19:17:Process mdt00_000 (pid: 20284, threadinfo ffff88005ab4c000, task ffff880060746aa0)
11:19:17:Stack:
11:19:17: ffff880002203f48 ffff8800022115a0 000000000000000a ffff88005ab4dfd8
11:19:17:<d> ffff88005ab4dfd8 0000000000011400 ffffffff81096c50 0000000000000046
11:19:17:<d> ffff88005ab4dfd8 ffff880026d28028 ffff880026d28000 0000000000000002
11:19:17:Call Trace:
11:19:17: <IRQ> 
11:19:17: [<ffffffff81096c50>] ? hrtimer_interrupt+0x140/0x250
11:19:17: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
11:19:17: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
11:19:17: [<ffffffff81073ca5>] ? irq_exit+0x85/0x90
11:19:17: [<ffffffff81506050>] ? smp_apic_timer_interrupt+0x70/0x9b
11:19:17: [<ffffffff8100bc13>] ? apic_timer_interrupt+0x13/0x20
11:19:17: <EOI> 
11:19:17: [<ffffffff8127e652>] ? memmove+0x42/0x1a0
11:19:18: [<ffffffffa0d89648>] ? iam_insert_key+0x68/0xb0 [osd_ldiskfs]
11:19:18: [<ffffffffa0d896d0>] ? iam_insert_key_lock+0x40/0x50 [osd_ldiskfs]
11:19:18: [<ffffffffa0d8ca3d>] ? iam_lfix_split+0x12d/0x150 [osd_ldiskfs]
11:19:18: [<ffffffffa0d8bedd>] ? iam_it_rec_insert+0x20d/0x300 [osd_ldiskfs]
11:19:18: [<ffffffffa0d8c071>] ? iam_insert+0xa1/0xb0 [osd_ldiskfs]
11:19:18: [<ffffffffa0d87717>] ? osd_oi_insert+0x1e7/0x5b0 [osd_ldiskfs]
11:19:18: [<ffffffffa0d7b0e5>] ? __osd_oi_insert+0x145/0x1e0 [osd_ldiskfs]
11:19:18: [<ffffffffa0d7ff68>] ? osd_object_ea_create+0x1d8/0x460 [osd_ldiskfs]
11:19:18: [<ffffffffa0aad28c>] ? mdd_object_create_internal+0x13c/0x2a0 [mdd]
11:19:18: [<ffffffffa0ace2fa>] ? mdd_create+0x16ba/0x20c0 [mdd]
11:19:18: [<ffffffffa0d7df8f>] ? osd_xattr_get+0x9f/0x360 [osd_ldiskfs]
11:19:18: [<ffffffffa0941637>] ? cml_create+0x97/0x250 [cmm]
11:19:18: [<ffffffffa0b37ddf>] ? mdt_version_get_save+0x8f/0xd0 [mdt]
11:19:18: [<ffffffffa0b4bb9f>] ? mdt_reint_open+0x108f/0x18a0 [mdt]
11:19:18: [<ffffffffa0ad400e>] ? md_ucred+0x1e/0x60 [mdd]
11:19:18: [<ffffffffa0b19235>] ? mdt_ucred+0x15/0x20 [mdt]
11:19:18: [<ffffffffa0b35151>] ? mdt_reint_rec+0x41/0xe0 [mdt]
11:19:18: [<ffffffffa0b2e9aa>] ? mdt_reint_internal+0x50a/0x810 [mdt]
11:19:18: [<ffffffffa0b2ef7d>] ? mdt_intent_reint+0x1ed/0x500 [mdt]
11:19:18: [<ffffffffa0b2b191>] ? mdt_intent_policy+0x371/0x6a0 [mdt]
11:19:18: [<ffffffffa04ff881>] ? ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc]
11:19:18: [<ffffffffa05279ef>] ? ldlm_handle_enqueue0+0x48f/0xf70 [ptlrpc]
11:19:18: [<ffffffffa0b2b506>] ? mdt_enqueue+0x46/0x130 [mdt]
11:19:18: [<ffffffffa0b22802>] ? mdt_handle_common+0x922/0x1740 [mdt]
11:19:18: [<ffffffffa0b236f5>] ? mdt_regular_handle+0x15/0x20 [mdt]
11:19:18: [<ffffffffa05579cd>] ? ptlrpc_server_handle_request+0x40d/0xea0 [ptlrpc]
11:19:18: [<ffffffffa03bd65e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
11:19:18: [<ffffffffa054ef67>] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc]
11:19:18: [<ffffffff81060250>] ? default_wake_function+0x0/0x20
11:19:18: [<ffffffffa0558fb9>] ? ptlrpc_main+0xb59/0x1860 [ptlrpc]
11:19:18: [<ffffffffa0558460>] ? ptlrpc_main+0x0/0x1860 [ptlrpc]
11:19:18: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
11:19:18: [<ffffffffa0558460>] ? ptlrpc_main+0x0/0x1860 [ptlrpc]
11:19:18: [<ffffffffa0558460>] ? ptlrpc_main+0x0/0x1860 [ptlrpc]
11:19:18: [<ffffffff8100c140>] ? child_rip+0x0/0x20

Issue Links

• duplicates: LU-1881 sanity test 116 soft lockup