[LU-1865] sanity test 71 hangs

Mon, 29 May 2017 05:14:35 +0000

The test seems to hang and the OSS's console show below crash

Lustre: DEBUG MARKER: == sanity test 71: Running dbench on lustre (don't segment fault) ====== 22:18:21 (1347081501)
skb_over_panic: text:ffffffffa0109de1 len:1514 put:1514 head:(null) data:(null) tail:0x5ea end:0x0 dev:<NULL>
BUG: scheduling while atomic: swapper/0/0x10000100
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 0, comm: swapper Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff810387cb>] [<ffffffff810387cb>] native_safe_halt+0xb/0x10
RSP: 0018:ffffffff81a01ec8 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81a01ec8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81dda228
RBP: ffffffff8100ba4e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81099015
R13: ffffffff81a01e58 R14: ffffffff81012b89 R15: ffffffff81a01e28
FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f81685bc000 CR3: 000000007d34d000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a8d020)
Stack:
ffffffff81a01ee8 ffffffff810149cd ffffffff81a01fd8 ffffffff81c017c0
<d> ffffffff81a01f18 ffffffff81009e06 6db6db6db6db6db7 4bf3cc9e6401a325
<d> 0000000000000000 6db6db6db6db6db7 ffffffff81a01f28 ffffffff814e47aa
Call Trace:
[<ffffffff810149cd>] ? default_idle+0x4d/0xb0
[<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
[<ffffffff814e47aa>] ? rest_init+0x7a/0x80
[<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
[<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
[<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
Code: 55 48 89 e5 0f 1f 44 00 00 fa c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 fb c9 c3 0f 1f 40 00 55 48 89 e5 0f 1f 44 00 00 fb f4 <c9> c3 0f 1f 00 55 48 89 e5 0f 1f 44 00 00 f4 c9 c3 0f 1f 40 00
Call Trace:
[<ffffffff810149cd>] ? default_idle+0x4d/0xb0
[<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
[<ffffffff814e47aa>] ? rest_init+0x7a/0x80
[<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
[<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
[<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
-----------~~[ cut here ]~~-----------
kernel BUG at net/core/skbuff.c:127!
invalid opcode: 0000 1 SMP
last sysfs file: /sys/devices/system/cpu/possible
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 0, comm: swapper Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8142eb8d>] [<ffffffff8142eb8d>] skb_over_panic+0x5d/0x70
RSP: 0018:ffff880002203d80 EFLAGS: 00010286
RAX: 0000000000000074 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246
RBP: ffff880002203da0 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000005 R12: 0000000000000000
R13: 000000000000000c R14: ffff88006fea20c0 R15: 00000000000000c0
FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f8a50f6a230 CR3: 000000006fc08000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a8d020)
Stack:
00000000000005ea 0000000000000000 ffffffff817bebd7 00000000000000c0
<d> ffff880002203dc0 ffffffff8142fbcc ffff88003768a6e0 00000000320005ee
<d> ffff880002203e60 ffffffffa0109de1 ffff880002203e40 0000000000000600
Call Trace:
<IRQ>
[<ffffffff8142fbcc>] skb_put+0x5c/0x70
[<ffffffffa0109de1>] cp_rx_poll+0x231/0x4f0 [8139cp]
[<ffffffff8143f603>] net_rx_action+0x103/0x2f0
[<ffffffff81073ec1>] __do_softirq+0xc1/0x1e0
[<ffffffff8100c24c>] call_softirq+0x1c/0x30
[<ffffffff8100de85>] do_softirq+0x65/0xa0
[<ffffffff81073ca5>] irq_exit+0x85/0x90
[<ffffffff81505f65>] do_IRQ+0x75/0xf0
[<ffffffff8100ba53>] ret_from_intr+0x0/0x11
<EOI>
[<ffffffff810387cb>] ? native_safe_halt+0xb/0x10
[<ffffffff810149cd>] default_idle+0x4d/0xb0
[<ffffffff81009e06>] cpu_idle+0xb6/0x110
[<ffffffff814e47aa>] rest_init+0x7a/0x80
[<ffffffff81c21f7b>] start_kernel+0x424/0x430
[<ffffffff81c2133a>] x86_64_start_reservations+0x125/0x129
[<ffffffff81c21438>] x86_64_start_kernel+0xfa/0x109
Code: 8b 57 68 48 89 44 24 10 8b 87 cc 00 00 00 48 89 44 24 08 8b bf c8 00 00 00 31 c0 48 89 3c 24 48 c7 c7 f8 65 80 81 e8 c5 ea 0c 00 <0f> 0b eb fe 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48
RIP [<ffffffff8142eb8d>] skb_over_panic+0x5d/0x70
RSP <ffff880002203d80>
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Aug 28 14:50:57 PDT 2012

Whamcloud Community JIRA

[LU-1865] sanity test 71 hangs