[LU-1980] stack overran on mds when use zfs on ost

Mon, 29 May 2017 05:47:23 +0000

Run compilebench on lab's VM with USE_OFD=yes, OSTFSTYPE=zfs

Lustre: DEBUG MARKER: /usr/sbin/lctl mark == parallel-scale test compilebench: compilebench == 21:26:31 (1348028791)
Lustre: DEBUG MARKER: == parallel-scale test compilebench: compilebench == 21:26:31 (1348028791)
Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt
Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 4 -r 4 --makej
Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 4 -r 4 --makej
[root@client-12vm3 ~]#
[root@client-12vm3 ~]#
[root@client-12vm3 ~]#
[root@client-12vm3 ~]# uptime
22:45:30 up 1:23, 1 user, load average: 0.00, 0.00, 0.00
[root@client-12vm3 ~]# BUG: unable to handle kernel paging request at 0000000000019288
IP: [<ffffffff8105422c>] update_curr+0x14c/0x1f0
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0002 1 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8105422c>] [<ffffffff8105422c>] update_curr+0x14c/0x1f0
RSP: 0018:ffff880002203db8 EFLAGS: 00010006
RAX: ffff880037458ae0 RBX: ffffffffffff8800 RCX: ffff88007faa60c0
RDX: 0000000000019288 RSI: ffff88004750d538 RDI: ffff880037458b18
RBP: ffff880002203de8 R08: ffffffff8160b725 R09: 0000000000000000
R10: 0000000000000010 R11: 0000000000000000 R12: ffff880002216728
R13: 00000000000f5021 R14: 000005038fe26d29 R15: ffff880037458ae0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000019288 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880002203dc8 0000000000000000 ffff880037458b18 ffff880002216728
<d> 0000000000000000 0000000000000000 ffff880002203e18 ffffffff810547db
<d> ffff8800022166c0 0000000000000000 00000000000166c0 0000000000000000
Call Trace:
<IRQ>
[<ffffffff810547db>] task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 2 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff8800022038c8 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002203df0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002203928 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002203e18
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002203e18 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff880002203db8
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 3 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff8800022033d8 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff8800022038f0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002203438 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002203928
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002203928 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff8800022038c8
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 4 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002202ee8 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002203400 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002202f48 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002203438
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002203438 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff8800022033d8
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 5 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff8800022029f8 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002202f10 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002202a58 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002202f48
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002202f48 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff880002202ee8
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 6 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002202508 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002202a20 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002202568 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002202a58
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002202a58 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff8800022029f8
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 7 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002202018 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002202530 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002202078 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002202568
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002202568 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff880002202508
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
[<ffffffff810547db>] ? task_tick_fair+0xdb/0x160
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 8 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002201b28 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff880002202040 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002201b88 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002202078
R13: ffff8800560f4000 R14: ffffffff81600480 R15: ffff880002203fc0
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002202078 ffff880002201fc0
<d> ffffffff8177dca8 ffffffff810547db ffffffff817ca485 ffff880002202018
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 ffff880002203fc0
Call Trace:
<IRQ>
<EOI>
[<ffffffff8127e64a>] ? memmove+0x3a/0x1a0
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 9 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002201638 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff8800560f5478 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff880002201698 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002201c28
R13: ffff8800560f4000 R14: ffffffff81600480 R15: 0000000000000000
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002201c28 ffffffffffffe000
<d> ffffffff8177dca8 ffffffff8127e64a ffffffff8177dcd4 ffff8800560f5420
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 0000000000000000
Call Trace:
<IRQ>
<EOI>
[<ffffffff8127e64a>] ? memmove+0x3a/0x1a0
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 10 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8100f5ad>] [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
RSP: 0018:ffff880002201148 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff8800560f5478 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
RBP: ffff8800022011a8 R08: ffffffff81c019c0 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000001 R12: ffff880002201738
R13: ffff8800560f4000 R14: ffffffff81600480 R15: 0000000000000000
FS: 00007f8aea604700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000009e8 CR3: 0000000070b7b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mdt00_003 (pid: 4524, threadinfo ffff8800560f4000, task ffff880037458ae0)
Stack:
ffff880000000018 ffff8800560f5ff8 ffff880002201738 ffffffffffffe000
<d> ffffffff8177dca8 ffffffff8127e64a ffffffff8177dcd4 ffff8800560f5420
<d> 000000000000cbe0 ffffffff81600480 ffffffff8177dca8 0000000000000000
Call Trace:
<IRQ>
<EOI>
[<ffffffff8127e64a>] ? memmove+0x3a/0x1a0
BUG: unable to handle kernel NULL pointer dereference at 00000000000009e8
IP: [<ffffffff8100f5ad>] print_context_stack+0xad/0x140
PGD 70b7c067 PUD 70b7d067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 11 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/speed
CPU 0
Modules linked in: lustre(U) ofd(U) ost(U) cmm(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mds(U) mgs(U) osd_zfs(U) jbd2 obdecho(U) mgc(U) lquota(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4524, comm: mdt00_003 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 #1
Call Trace:
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-279.5.1.el6_lustre.g293c36b.x86_64 (jenkins@builder-1-sde1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Aug 28 14:50:57 PDT 2012
Command line: ro root=UUID=8abfe7c9-bc15-4477-b48a-77a78939ab3d rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD console=ttyS0,115200 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off memmap=exactmap memmap=627K@4K memmap=131449K@49779K elfcorehdr=181228K memmap=4K$0K memmap=9K$631K memmap=64K$960K memmap=12K$2097140K memmap=272K$4194032K
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
Disabled fast string operations
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000100 - 000000000009dc00 (usable)
BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007fffd000 (usable)
BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved)
BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)
last_pfn = 0x7fffd max_arch_pfn = 0x400000000
user-defined physical RAM map:
user: 0000000000000000 - 0000000000001000 (reserved)
user: 0000000000001000 - 000000000009dc00 (usable)
user: 000000000009dc00 - 00000000000a0000 (reserved)
user: 00000000000f0000 - 0000000000100000 (reserved)
user: 000000000309cc00 - 000000000b0fb000 (usable)
user: 000000007fffd000 - 0000000080000000 (reserved)
user: 00000000fffbc000 - 0000000100000000 (reserved)
DMI 2.4 present.
SMBIOS version 2.4 @ 0xFDA50
last_pfn = 0xb0fb max_arch_pfn = 0x400000000
PAT not supported by CPU.
init_memory_mapping: 0000000000000000-000000000b0fb000
RAMDISK: 0acac000 - 0b0ee480
ACPI: RSDP 00000000000fda20 00014 (v00 BOCHS )
ACPI: RSDT 000000007fffd740 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
ACPI: FACP 000000007ffffe40 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 000000007fffd900 024FD (v01 BXPC BXDSDT 00000001 INTL 20090123)
ACPI: FACS 000000007ffffe00 00040
ACPI: SSDT 000000007fffd860 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
ACPI: APIC 000000007fffd770 00078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
No NUMA configuration found
Faking a node at 0000000000000000-000000000b0fb000
Bootmem setup node 0 0000000000000000-000000000b0fb000
NODE_DATA [0000000000009000 - 000000000003cfff]
bootmap [000000000003d000 - 000000000003e61f] pages 2
(7 early reservations) ==> bootmem [0000000000 - 000b0fb000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#2 [0004000000 - 0005012024] TEXT DATA BSS ==> [0004000000 - 0005012024]
#3 [000acac000 - 000b0ee480] RAMDISK ==> [000acac000 - 000b0ee480]
#4 [000009dc00 - 0000100000] BIOS reserved ==> [000009dc00 - 0000100000]
#5 [0005013000 - 0005013081] BRK ==> [0005013000 - 0005013081]
#6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff8800000fda70] fda70
kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr 0:4c1f641, boot clock
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x0000009d
0: 0x0000309d -> 0x0000b0fb
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 1 CPUs, 0 hotplug CPUs
PM: Registered nosave memory: 000000000009d000 - 000000000009e000
PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
PM: Registered nosave memory: 0000000000100000 - 000000000309d000
Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000)

Whamcloud Community JIRA

[LU-1980] stack overran on mds when use zfs on ost