[LU-2169] ZFS sanity test 64b panic

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Minor
• Reporter: Oleg Drokin
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

Running FSTYPE=zfs SLOW=yes REFORMAT=yes ONLY="64b 77" sh sanity.sh I hit this panic 5 seconds into the run:

[ 3845.859344] Lustre: DEBUG MARKER: == sanity test 64b: check out-of-space dete
ction on client ============= 14:53:17 (1350154397)
[ 3850.298795] BUG: unable to handle kernel paging request at ffff88020b506000
[ 3850.301381] IP: [<ffffffff8127e0db>] memcpy+0xb/0x120
[ 3850.302187] PGD 1a26063 PUD 288d067 PMD 28e8067 PTE 800000020b506160
[ 3850.302279] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC[ 3850.302279] last sysfs file: /sys/devices/system/cpu/possible
[ 3850.302279] CPU 4
[ 3850.302279] Modules linked in: lustre ofd osp lod ost mdt mdd mds mgs osd_zfs
 lquota obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass lvfs ksocklnd lnet l
ibcfs ext2 zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate jbd
 sha512_generic sha256_generic sunrpc ipv6 microcode virtio_balloon virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pat
a_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloade
d: libcfs]
[ 3850.302279]
[ 3850.302279] Pid: 27858, comm: ll_ost_io01_002 Tainted: P    B      ----------
-----    2.6.32-debug #5 Bochs Bochs
[ 3850.302279] RIP: 0010:[<ffffffff8127e0db>]  [<ffffffff8127e0db>] memcpy+0xb/0
x120
[ 3850.302279] RSP: 0018:ffff8801c6297508  EFLAGS: 00010246
[ 3850.302279] RAX: ffff88020b506000 RBX: 0000000000000000 RCX: 0000000000000200
[ 3850.302279] RDX: 0000000000000000 RSI: ffff880124dfb000 RDI: ffff88020b506000
[ 3850.302279] RBP: ffff8801c6297570 R08: 0000000000001000 R09: ffff8802958a6a00
[ 3850.302279] R10: 0000000000000000 R11: ffff88020b506000 R12: 0000000000041000
[ 3850.302279] R13: 0000000000001000 R14: 0000000000000000 R15: ffff880124dfb000
[ 3850.302279] FS:  00007eff7e8b8700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
[ 3850.302279] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3850.302279] CR2: ffff88020b506000 CR3: 0000000193751000 CR4: 00000000000006e0
[ 3850.302279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3850.302279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 3850.302279] Process ll_ost_io01_002 (pid: 27858, threadinfo ffff8801c6296000, task ffff880175bfe2c0)
[ 3850.302279] Stack:
[ 3850.302279]  ffffffffa08f20f3 0000000000001000 ffff88020b506000 ffff880143696a00
[ 3850.302279] <d> ffff8802958a6a00 ffffea0007279950 0000004200000042 ffff8801c6297580
[ 3850.302279] <d> ffff880202821df0 ffff88025672fdf0 ffff88021669edf0 ffff88021669edf0
[ 3850.302279] Call Trace:
[ 3850.302279]  [<ffffffffa08f20f3>] ? lnet_copy_kiov2kiov+0x153/0x450 [lnet]
[ 3850.302279]  [<ffffffffa08f6a72>] lolnd_recv+0xd2/0xe0 [lnet]
[ 3850.302279]  [<ffffffffa08ef0db>] lnet_ni_recv+0xbb/0x380 [lnet]
[ 3850.302279]  [<ffffffffa08f6378>] lnet_parse+0x1838/0x1900 [lnet]
[ 3850.302279]  [<ffffffffa08f6aab>] lolnd_send+0x2b/0xb0 [lnet]
[ 3850.302279]  [<ffffffffa08eef5b>] lnet_ni_send+0x4b/0x110 [lnet]
[ 3850.302279]  [<ffffffffa08f36d1>] lnet_send+0x851/0xc10 [lnet]
[ 3850.302279]  [<ffffffffa08f59cd>] lnet_parse+0xe8d/0x1900 [lnet]
[ 3850.302279]  [<ffffffffa0c9534f>] ? cfs_trace_unlock_tcd+0x3f/0xa0 [libcfs]
[ 3850.302279]  [<ffffffffa08f6aab>] lolnd_send+0x2b/0xb0 [lnet]
[ 3850.302279]  [<ffffffffa08eef5b>] lnet_ni_send+0x4b/0x110 [lnet]
[ 3850.302279]  [<ffffffffa08f36d1>] lnet_send+0x851/0xc10 [lnet]
[ 3850.302279]  [<ffffffffa08f3d74>] LNetGet+0x2e4/0x830 [lnet]
[ 3850.302279]  [<ffffffffa0f36370>] ptlrpc_start_bulk_transfer+0x160/0x650 [ptlrpc]
[ 3850.302279]  [<ffffffffa0f07830>] target_bulk_io+0x180/0x930 [ptlrpc]
[ 3850.302279]  [<ffffffffa0ca66d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[ 3850.302279]  [<ffffffffa0c968b5>] ? cfs_waitq_init+0x15/0x20 [libcfs]
[ 3850.302279]  [<ffffffffa0f2cf5e>] ? new_bulk+0x10e/0x220 [ptlrpc]
[ 3850.302279]  [<ffffffffa0d5adb1>] ? class_export_get+0x81/0x90 [obdclass]
[ 3850.302279]  [<ffffffffa0f29ce8>] ? __ptlrpc_prep_bulk_page+0x68/0x1a0 [ptlrpc]
[ 3850.302279]  [<ffffffffa03de45f>] ost_brw_write+0x132f/0x15d0 [ost]
[ 3850.302279]  [<ffffffffa0ca66d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[ 3850.302279]  [<ffffffffa03e3250>] ost_handle+0x3120/0x4550 [ost]
[ 3850.302279]  [<ffffffffa0ca2464>] ? libcfs_id2str+0x74/0xb0 [libcfs]
[ 3850.302279]  [<ffffffffa0f4a483>] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc]
[ 3850.302279]  [<ffffffffa0c9666e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[ 3850.302279]  [<ffffffffa0f43171>] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc]
[ 3850.302279]  [<ffffffffa0ca66d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[ 3850.302279]  [<ffffffff81051f73>] ? __wake_up+0x53/0x70
[ 3850.302279]  [<ffffffffa0f4d01a>] ptlrpc_main+0xb9a/0x1960 [ptlrpc]
[ 3850.302279]  [<ffffffffa0f4c480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[ 3850.302279]  [<ffffffff8100c14a>] child_rip+0xa/0x20
[ 3850.302279]  [<ffffffffa0f4c480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[ 3850.302279]  [<ffffffffa0f4c480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[ 3850.302279]  [<ffffffff8100c140>] ? child_rip+0x0/0x20
[ 3850.302279] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c

Comments

[Oleg Drokin]

Just had this one happen again.
It seems to only happen after you had previous sanity.sh run without reboot, so perhaps some state leftover from before spoils things later?

[25311.467375] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 03:12:12 (1350198732)
[25311.742484] BUG: unable to handle kernel paging request at ffff8800bd658000
[25311.744474] IP: [<ffffffff8127e0db>] memcpy+0xb/0x120
[25311.746280] PGD 1a26063 PUD 501067 PMD 6ed067 PTE 80000000bd658160
[25311.746280] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[25311.746280] last sysfs file: /sys/devices/pci0000:00/0000:00:06.0/local_cpus
[25311.746280] CPU 6
[25311.746280] Modules linked in: lustre ofd osp lod ost mdt mdd mds mgs osd_zfs
 lquota obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass lvfs ksocklnd lnet l
ibcfs ext2 zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate jbd
 sha512_generic sha256_generic sunrpc ipv6 microcode virtio_balloon virtio_net i
2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
[25311.746280]
[25311.746280] Pid: 25007, comm: ll_ost_io01_003 Tainted: P    B      ---------------    2.6.32-debug #5 Bochs Bochs
[25311.746280] RIP: 0010:[<ffffffff8127e0db>]  [<ffffffff8127e0db>] memcpy+0xb/0x120
[25311.746280] RSP: 0018:ffff88009fe4d508  EFLAGS: 00010246
[25311.746280] RAX: ffff8800bd658000 RBX: 0000000000000000 RCX: 0000000000000200
[25311.746280] RDX: 0000000000000000 RSI: ffff88004bfae000 RDI: ffff8800bd658000
[25311.746280] RBP: ffff88009fe4d570 R08: 0000000000001000 R09: ffff8800560c8a20
[25311.746280] R10: 0000000000000000 R11: ffff8800bd658000 R12: 000000000003f000
[25311.746280] R13: 0000000000001000 R14: 0000000000000000 R15: ffff88004bfae000
[25311.746280] FS:  00007f9a3b65d700(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
[25311.746280] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[25311.746280] CR2: ffff8800bd658000 CR3: 0000000001a25000 CR4: 00000000000006e0
[25311.746280] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[25311.746280] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[25311.746280] Process ll_ost_io01_003 (pid: 25007, threadinfo ffff88009fe4c000, task ffff8800bd838100)
[25311.746280] Stack:
[25311.746280]  ffffffffa07ba0f3 0000000000001000 ffff8800bd658000 ffff880108b89a20
[25311.746280] <d> ffff8800560c8a20 ffffea000296e340 0000004000000040 ffff88009fe4d580
[25311.746280] <d> ffff8800dd5a1df0 ffff880282e7edf0 ffff88018fff0df0 ffff88018fff0df0
[25311.746280] Call Trace:
[25311.746280]  [<ffffffffa07ba0f3>] ? lnet_copy_kiov2kiov+0x153/0x450 [lnet]
[25311.746280]  [<ffffffffa07bea72>] lolnd_recv+0xd2/0xe0 [lnet]
[25311.746280]  [<ffffffffa07b70db>] lnet_ni_recv+0xbb/0x380 [lnet]
[25311.746280]  [<ffffffffa07be378>] lnet_parse+0x1838/0x1900 [lnet]
[25311.746280]  [<ffffffffa07beaab>] lolnd_send+0x2b/0xb0 [lnet]
[25311.746280]  [<ffffffffa07b6f5b>] lnet_ni_send+0x4b/0x110 [lnet]
[25311.746280]  [<ffffffffa07bb6d1>] lnet_send+0x851/0xc10 [lnet]
[25311.746280]  [<ffffffffa07bd9cd>] lnet_parse+0xe8d/0x1900 [lnet]
[25311.746280]  [<ffffffffa072c34f>] ? cfs_trace_unlock_tcd+0x3f/0xa0 [libcfs]
[25311.746280]  [<ffffffffa07beaab>] lolnd_send+0x2b/0xb0 [lnet]
[25311.746280]  [<ffffffffa07b6f5b>] lnet_ni_send+0x4b/0x110 [lnet]
[25311.746280]  [<ffffffffa07bb6d1>] lnet_send+0x851/0xc10 [lnet]
[25311.746280]  [<ffffffffa07bbd74>] LNetGet+0x2e4/0x830 [lnet]
[25311.746280]  [<ffffffffa0e44370>] ptlrpc_start_bulk_transfer+0x160/0x650 [ptlrpc]
[25311.746280]  [<ffffffffa0e15830>] target_bulk_io+0x180/0x930 [ptlrpc]
[25311.746280]  [<ffffffffa073d6d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[25311.746280]  [<ffffffffa072d8b5>] ? cfs_waitq_init+0x15/0x20 [libcfs]
[25311.746280]  [<ffffffffa0e3af5e>] ? new_bulk+0x10e/0x220 [ptlrpc]
[25311.746280]  [<ffffffffa0ba0db1>] ? class_export_get+0x81/0x90 [obdclass]
[25311.746280]  [<ffffffffa0e37ce8>] ? __ptlrpc_prep_bulk_page+0x68/0x1a0 [ptlrpc]
[25311.746280]  [<ffffffffa083045f>] ost_brw_write+0x132f/0x15d0 [ost]
[25311.746280]  [<ffffffffa073d6d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[25311.746280]  [<ffffffffa0835250>] ost_handle+0x3120/0x4550 [ost]
[25311.746280]  [<ffffffffa0739464>] ? libcfs_id2str+0x74/0xb0 [libcfs]
[25311.746280]  [<ffffffffa0e58483>] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc]
[25311.746280]  [<ffffffffa072d66e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[25311.746280]  [<ffffffffa0e51171>] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc]
[25311.746280]  [<ffffffffa073d6d1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
[25311.746280]  [<ffffffff81051f73>] ? __wake_up+0x53/0x70
[25311.746280]  [<ffffffffa0e5b01a>] ptlrpc_main+0xb9a/0x1960 [ptlrpc]
[25311.746280]  [<ffffffffa0e5a480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[25311.746280]  [<ffffffff8100c14a>] child_rip+0xa/0x20
[25311.746280]  [<ffffffffa0e5a480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[25311.746280]  [<ffffffffa0e5a480>] ? ptlrpc_main+0x0/0x1960 [ptlrpc]
[25311.746280]  [<ffffffff8100c140>] ? child_rip+0x0/0x20
[25311.746280] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c
[25311.746280] RIP  [<ffffffff8127e0db>] memcpy+0xb/0x120
[25311.746280]  RSP <ffff88009fe4d508>
[25311.746280] CR2: ffff8800bd658000
[25311.746280] ---[ end trace 96e412e2254363a4 ]---

[Oleg Drokin]

I now hit this very reliably in sanity test 224b with no zfs in picture.

[Oleg Drokin]

new trace looks like this:

<4>[11515.961133] Lustre: lustre-OST0001: Client 6c55d310-d816-6361-9358-41f6556f3804 (at 0@lo) reconnecting
<3>[11516.960148] LustreError: 7151:0:(ldlm_lib.c:2701:target_bulk_io()) @@@ Reconnect on bulk PUT  req@ffff8800b676abe8 x1467459440236164/t0(0) o3->6c55d310-d816-6361-9358-41f6556f3804@0@lo:0/0 lens 488/432 e 1 to 0 dl 1399478720 ref 1 fl Interpret:/0/0 rc 0/0
<1>[11518.960326] BUG: unable to handle kernel paging request at ffff880080b35000
<1>[11518.960635] IP: [<ffffffff8128c91b>] memcpy+0xb/0x120
<4>[11518.961004] PGD 1a26063 PUD 501067 PMD 507067 PTE 8000000080b35060
<4>[11518.961313] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
<4>[11518.961580] last sysfs file: /sys/devices/system/cpu/possible
<4>[11518.961855] CPU 2 
<4>[11518.961893] Modules linked in: lustre ofd osp lod ost mdt mdd mgs nodemap osd_ldiskfs ldiskfs lquota lfsck mgc lov osc mdc lmv fid fld ptlrpc obdclass ksocklnd lnet libcfs ext2 exportfs jbd sha512_generic sha256_generic ext4 jbd2 mbcache ppdev parport_pc parport virtio_balloon virtio_console i2c_piix4 i2c_core virtio_net virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache auth_rpcgss nfs_acl sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: obdecho]
<4>[11518.964196] 
<4>[11518.964196] Pid: 11253, comm: ll_ost_io01_006 Tainted: G        W  ---------------    2.6.32-rhe6.5-debug #2 Bochs Bochs
<4>[11518.964196] RIP: 0010:[<ffffffff8128c91b>]  [<ffffffff8128c91b>] memcpy+0xb/0x120
<4>[11518.964196] RSP: 0018:ffff88003e7b5738  EFLAGS: 00010246
<4>[11518.964196] RAX: ffff8800aa7b2000 RBX: 0000000000000000 RCX: 0000000000000200
<4>[11518.964196] RDX: 0000000000000000 RSI: ffff880080b35000 RDI: ffff8800aa7b2000
<4>[11518.964196] RBP: ffff88003e7b57a0 R08: 0000000000001000 R09: ffff8800b7677370
<4>[11518.964196] R10: 0000000000000000 R11: ffff8800aa7b2000 R12: 0000000000000000
<4>[11518.964196] R13: 0000000000001000 R14: 0000000000000000 R15: ffff880080b35000
<4>[11518.964196] FS:  0000000000000000(0000) GS:ffff880006300000(0000) knlGS:0000000000000000
<4>[11518.964196] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>[11518.964196] CR2: ffff880080b35000 CR3: 0000000001a25000 CR4: 00000000000006e0
<4>[11518.964196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[11518.964196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[11518.964196] Process ll_ost_io01_006 (pid: 11253, threadinfo ffff88003e7b4000, task ffff8800a92fa500)
<4>[11518.964196] Stack:
<4>[11518.964196]  ffffffffa0be7963 0000000000001000 ffff8800aa7b2000 ffff88009f149b68
<4>[11518.964196] <d> ffff8800b7677370 ffffea000254aef0 0000000100000001 0000000000000000
<4>[11518.964196] <d> ffff8800552a1df0 ffff880057507df0 ffff8800268f3df0 ffff8800268f3df0
<4>[11518.964196] Call Trace:
<4>[11518.964196]  [<ffffffffa0be7963>] ? lnet_copy_kiov2kiov+0x153/0x410 [lnet]
<4>[11518.964196]  [<ffffffffa0bec472>] lolnd_recv+0xd2/0xe0 [lnet]
<4>[11518.964196]  [<ffffffff8151677e>] ? _spin_unlock+0xe/0x10
<4>[11518.964196]  [<ffffffffa0be4f7b>] lnet_ni_recv+0xbb/0x320 [lnet]
<4>[11518.964196]  [<ffffffffa0be5263>] lnet_recv_put+0x83/0xb0 [lnet]
<4>[11518.964196]  [<ffffffffa0beb7f4>] lnet_parse+0x1254/0x18c0 [lnet]
<4>[11518.964196]  [<ffffffffa0bec4ab>] lolnd_send+0x2b/0xa0 [lnet]
<4>[11518.964196]  [<ffffffffa0be4e1b>] lnet_ni_send+0x4b/0xf0 [lnet]
<4>[11518.964196]  [<ffffffffa0be9200>] lnet_send+0x850/0xb70 [lnet]
<4>[11518.964196]  [<ffffffffa0bea05a>] LNetPut+0x31a/0x860 [lnet]
<4>[11518.964196]  [<ffffffffa0701487>] ptlrpc_start_bulk_transfer+0x1e7/0x720 [ptlrpc]
<4>[11518.964196]  [<ffffffff81516894>] ? _spin_lock_irqsave+0x24/0x30
<4>[11518.964196]  [<ffffffff8108292b>] ? try_to_del_timer_sync+0x7b/0xe0
<4>[11518.964196]  [<ffffffffa06d0318>] target_bulk_io+0x6e8/0xa00 [ptlrpc]
<4>[11518.964196]  [<ffffffff81514239>] ? schedule_timeout+0x199/0x2e0
<4>[11518.964196]  [<ffffffffa076a3e5>] tgt_brw_read+0x10e5/0x1150 [ptlrpc]
<4>[11518.964196]  [<ffffffffa0706cad>] ? lustre_pack_reply_v2+0x1fd/0x2a0 [ptlrpc]
<4>[11518.964196]  [<ffffffff8105de00>] ? default_wake_function+0x0/0x20
<4>[11518.964196]  [<ffffffffa0706dfe>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
<4>[11518.964196]  [<ffffffffa076754c>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4>[11518.964196]  [<ffffffffa0717f88>] ptlrpc_main+0xcc8/0x1950 [ptlrpc]
<4>[11518.964196]  [<ffffffffa07172c0>] ? ptlrpc_main+0x0/0x1950 [ptlrpc]
<4>[11518.964196]  [<ffffffff81098c06>] kthread+0x96/0xa0
<4>[11518.964196]  [<ffffffff8100c24a>] child_rip+0xa/0x20
<4>[11518.964196]  [<ffffffff81098b70>] ? kthread+0x0/0xa0
<4>[11518.964196]  [<ffffffff8100c240>] ? child_rip+0x0/0x20
<4>[11518.964196] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 
<1>[11518.964196] RIP  [<ffffffff8128c91b>] memcpy+0xb/0x120
<4>[11518.964196]  RSP <ffff88003e7b5738>
<4>[11518.964196] CR2: ffff880080b35000

[Oleg Drokin]

Just had this happen again on a new setup, current master.
Test 224b again:

[10301.251331] BUG: unable to handle kernel paging request at ffff88029aa23000
[10301.252837] IP: [<ffffffff8138590d>] memcpy+0xd/0x110
[10301.253477] PGD 2e75067 PUD 33ebfa067 PMD 33eb24067 PTE 800000029aa23060
[10301.254185] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[10301.254823] Modules linked in: brd lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate mbcache jbd2 syscopyarea sysfillrect sysimgblt ttm drm_kms_helper ata_generic pata_acpi i2c_piix4 drm virtio_blk floppy pcspkr ata_piix serio_raw virtio_console virtio_balloon i2c_core libata nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs]
[10301.260919] CPU: 5 PID: 4882 Comm: ll_ost_io00_002 Tainted: P           OE  ------------   3.10.0-debug #2
[10301.262166] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[10301.262860] task: ffff8801aa070440 ti: ffff8802cc144000 task.ti: ffff8802cc144000
[10301.264050] RIP: 0010:[<ffffffff8138590d>]  [<ffffffff8138590d>] memcpy+0xd/0x110
[10301.265262] RSP: 0018:ffff8802cc147630  EFLAGS: 00010246
[10301.265898] RAX: ffff8802f8d55000 RBX: 0000000000001000 RCX: 0000000000000200
[10301.266563] RDX: 0000000000000000 RSI: ffff88029aa23000 RDI: ffff8802f8d55000
[10301.267255] RBP: ffff8802cc147688 R08: ffff8802f8d55000 R09: ffff88029aa23000
[10301.268146] R10: ffff88024abd5fe0 R11: ffff8802cf93ffe0 R12: 0000000000000000
[10301.269011] R13: 0000000000000000 R14: ffff8802f8d55000 R15: 0000000000000000
[10301.269878] FS:  0000000000000000(0000) GS:ffff88033e4a0000(0000) knlGS:0000000000000000
[10301.271249] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[10301.271681] CR2: ffff88029aa23000 CR3: 0000000001c0e000 CR4: 00000000000006e0
[10301.273318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[10301.273984] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[10301.274608] Stack:
[10301.275164]  ffffffffa03035d1 0000000100000001 ffff8802cf93ffe0 ffff88024abd5fe0
[10301.276335]  0000000000001000 ffff88029aa23000 ffff88023ac08e00 ffff880048b10e00
[10301.277496]  0000000000000000 ffff880048b10e00 ffff880048b10e00 ffff8802cc1476b0
[10301.279048] Call Trace:
[10301.279640]  [<ffffffffa03035d1>] ? lnet_copy_kiov2kiov+0x161/0x3e0 [lnet]
[10301.280324]  [<ffffffffa030ac71>] lolnd_recv+0xb1/0xc0 [lnet]
[10301.280976]  [<ffffffffa0305fb6>] lnet_ni_recv+0xc6/0x320 [lnet]
[10301.282541]  [<ffffffffa03066f1>] lnet_recv_put+0x81/0xb0 [lnet]
[10301.283254]  [<ffffffffa03088f6>] lnet_parse_local+0x5a6/0xd40 [lnet]
[10301.284530]  [<ffffffffa021abd8>] ? cfs_percpt_lock+0x58/0x110 [libcfs]
[10301.285273]  [<ffffffffa030993a>] lnet_parse+0x8aa/0xfa0 [lnet]
[10301.285936]  [<ffffffffa030ae0b>] lolnd_send+0x2b/0xa0 [lnet]
[10301.286611]  [<ffffffffa0301e0e>] lnet_ni_send+0x3e/0xd0 [lnet]
[10301.287301]  [<ffffffffa03071f7>] lnet_send+0x77/0x180 [lnet]
[10301.287962]  [<ffffffffa0307545>] LNetPut+0x245/0x7a0 [lnet]
[10301.288695]  [<ffffffffa09fc476>] ptlrpc_start_bulk_transfer+0x2d6/0x790 [ptlrpc]
[10301.289935]  [<ffffffff8139138b>] ? free_object+0x8b/0xb0
[10301.290594]  [<ffffffff8139138b>] ? free_object+0x8b/0xb0
[10301.291301]  [<ffffffffa09c0173>] target_bulk_io+0x823/0xb20 [ptlrpc]
[10301.292021]  [<ffffffffa0a686e3>] tgt_brw_read+0x773/0x1870 [ptlrpc]
[10301.292726]  [<ffffffff811cd4f9>] ? __kmalloc+0x649/0x660
[10301.293920]  [<ffffffff817063d7>] ? _raw_spin_unlock+0x27/0x40
[10301.294613]  [<ffffffff810f3b93>] ? is_module_address+0x23/0x30
[10301.295443]  [<ffffffff810e24ac>] ? static_obj+0x3c/0x50
[10301.304053]  [<ffffffff810b7cc0>] ? wake_up_state+0x20/0x20
[10301.304763]  [<ffffffffa0a65ccb>] tgt_request_handle+0x93b/0x1390 [ptlrpc]
[10301.305470]  [<ffffffffa0a10351>] ptlrpc_server_handle_request+0x251/0xae0 [ptlrpc]
[10301.306690]  [<ffffffffa0a14108>] ptlrpc_main+0xa58/0x1df0 [ptlrpc]
[10301.307383]  [<ffffffffa0a136b0>] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc]
[10301.308850]  [<ffffffff810a2eba>] kthread+0xea/0xf0
[10301.309462]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[10301.310126]  [<ffffffff8170fb98>] ret_from_fork+0x58/0x90
[10301.310746]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[10301.311400] Code: 43 4e 5b 5d c3 66 0f 1f 84 00 00 00 00 00 e8 fb fb ff ff eb e2 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 
[10301.342280] RIP  [<ffffffff8138590d>] memcpy+0xd/0x110
[10301.342945]  RSP <ffff8802cc147630>
[10301.343550] CR2: ffff88029aa23000