https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2169 Lustre <p>Running FSTYPE=zfs SLOW=yes REFORMAT=yes ONLY="64b 77" sh sanity.sh I hit this panic 5 seconds into the run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 3845.859344] Lustre: DEBUG MARKER: == sanity test 64b: check out-of-space dete ction on client ============= 14:53:17 (1350154397) [ 3850.298795] BUG: unable to handle kernel paging request at ffff88020b506000 [ 3850.301381] IP: [&lt;ffffffff8127e0db&gt;] memcpy+0xb/0x120 [ 3850.302187] PGD 1a26063 PUD 288d067 PMD 28e8067 PTE 800000020b506160 [ 3850.302279] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC[ 3850.302279] last sysfs file: /sys/devices/system/cpu/possible [ 3850.302279] CPU 4 [ 3850.302279] Modules linked in: lustre ofd osp lod ost mdt mdd mds mgs osd_zfs lquota obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass lvfs ksocklnd lnet l ibcfs ext2 zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate jbd sha512_generic sha256_generic sunrpc ipv6 microcode virtio_balloon virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pat a_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloade d: libcfs] [ 3850.302279] [ 3850.302279] Pid: 27858, comm: ll_ost_io01_002 Tainted: P B ---------- ----- 2.6.32-debug #5 Bochs Bochs [ 3850.302279] RIP: 0010:[&lt;ffffffff8127e0db&gt;] [&lt;ffffffff8127e0db&gt;] memcpy+0xb/0 x120 [ 3850.302279] RSP: 0018:ffff8801c6297508 EFLAGS: 00010246 [ 3850.302279] RAX: ffff88020b506000 RBX: 0000000000000000 RCX: 0000000000000200 [ 3850.302279] RDX: 0000000000000000 RSI: ffff880124dfb000 RDI: ffff88020b506000 [ 3850.302279] RBP: ffff8801c6297570 R08: 0000000000001000 R09: ffff8802958a6a00 [ 3850.302279] R10: 0000000000000000 R11: ffff88020b506000 R12: 0000000000041000 [ 3850.302279] R13: 0000000000001000 R14: 0000000000000000 R15: ffff880124dfb000 [ 3850.302279] FS: 00007eff7e8b8700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000 [ 3850.302279] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 3850.302279] CR2: ffff88020b506000 CR3: 0000000193751000 CR4: 00000000000006e0 [ 3850.302279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 3850.302279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 3850.302279] Process ll_ost_io01_002 (pid: 27858, threadinfo ffff8801c6296000, task ffff880175bfe2c0) [ 3850.302279] Stack: [ 3850.302279] ffffffffa08f20f3 0000000000001000 ffff88020b506000 ffff880143696a00 [ 3850.302279] &lt;d&gt; ffff8802958a6a00 ffffea0007279950 0000004200000042 ffff8801c6297580 [ 3850.302279] &lt;d&gt; ffff880202821df0 ffff88025672fdf0 ffff88021669edf0 ffff88021669edf0 [ 3850.302279] Call Trace: [ 3850.302279] [&lt;ffffffffa08f20f3&gt;] ? lnet_copy_kiov2kiov+0x153/0x450 [lnet] [ 3850.302279] [&lt;ffffffffa08f6a72&gt;] lolnd_recv+0xd2/0xe0 [lnet] [ 3850.302279] [&lt;ffffffffa08ef0db&gt;] lnet_ni_recv+0xbb/0x380 [lnet] [ 3850.302279] [&lt;ffffffffa08f6378&gt;] lnet_parse+0x1838/0x1900 [lnet] [ 3850.302279] [&lt;ffffffffa08f6aab&gt;] lolnd_send+0x2b/0xb0 [lnet] [ 3850.302279] [&lt;ffffffffa08eef5b&gt;] lnet_ni_send+0x4b/0x110 [lnet] [ 3850.302279] [&lt;ffffffffa08f36d1&gt;] lnet_send+0x851/0xc10 [lnet] [ 3850.302279] [&lt;ffffffffa08f59cd&gt;] lnet_parse+0xe8d/0x1900 [lnet] [ 3850.302279] [&lt;ffffffffa0c9534f&gt;] ? cfs_trace_unlock_tcd+0x3f/0xa0 [libcfs] [ 3850.302279] [&lt;ffffffffa08f6aab&gt;] lolnd_send+0x2b/0xb0 [lnet] [ 3850.302279] [&lt;ffffffffa08eef5b&gt;] lnet_ni_send+0x4b/0x110 [lnet] [ 3850.302279] [&lt;ffffffffa08f36d1&gt;] lnet_send+0x851/0xc10 [lnet] [ 3850.302279] [&lt;ffffffffa08f3d74&gt;] LNetGet+0x2e4/0x830 [lnet] [ 3850.302279] [&lt;ffffffffa0f36370&gt;] ptlrpc_start_bulk_transfer+0x160/0x650 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0f07830&gt;] target_bulk_io+0x180/0x930 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0ca66d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3850.302279] [&lt;ffffffffa0c968b5&gt;] ? cfs_waitq_init+0x15/0x20 [libcfs] [ 3850.302279] [&lt;ffffffffa0f2cf5e&gt;] ? new_bulk+0x10e/0x220 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0d5adb1&gt;] ? class_export_get+0x81/0x90 [obdclass] [ 3850.302279] [&lt;ffffffffa0f29ce8&gt;] ? __ptlrpc_prep_bulk_page+0x68/0x1a0 [ptlrpc] [ 3850.302279] [&lt;ffffffffa03de45f&gt;] ost_brw_write+0x132f/0x15d0 [ost] [ 3850.302279] [&lt;ffffffffa0ca66d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3850.302279] [&lt;ffffffffa03e3250&gt;] ost_handle+0x3120/0x4550 [ost] [ 3850.302279] [&lt;ffffffffa0ca2464&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [ 3850.302279] [&lt;ffffffffa0f4a483&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0c9666e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [ 3850.302279] [&lt;ffffffffa0f43171&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0ca66d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 3850.302279] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [ 3850.302279] [&lt;ffffffffa0f4d01a&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0f4c480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 3850.302279] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [ 3850.302279] [&lt;ffffffffa0f4c480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 3850.302279] [&lt;ffffffffa0f4c480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 3850.302279] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [ 3850.302279] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 &lt;f3&gt; 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c </pre> </div></div> LU-2169

ZFS sanity test 64b panic

Bug Minor Open Unresolved WC Triage Oleg Drokin Sat, 13 Oct 2012 14:59:27 +0000 Tue, 25 Jul 2017 22:49:41 +0000 Lustre 2.4.0 0 4 <p>Just had this one happen again.<br/> It seems to only happen after you had previous sanity.sh run without reboot, so perhaps some state leftover from before spoils things later?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[25311.467375] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block == 03:12:12 (1350198732) [25311.742484] BUG: unable to handle kernel paging request at ffff8800bd658000 [25311.744474] IP: [&lt;ffffffff8127e0db&gt;] memcpy+0xb/0x120 [25311.746280] PGD 1a26063 PUD 501067 PMD 6ed067 PTE 80000000bd658160 [25311.746280] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [25311.746280] last sysfs file: /sys/devices/pci0000:00/0000:00:06.0/local_cpus [25311.746280] CPU 6 [25311.746280] Modules linked in: lustre ofd osp lod ost mdt mdd mds mgs osd_zfs lquota obdecho mgc lov osc mdc lmv fid fld ptlrpc obdclass lvfs ksocklnd lnet l ibcfs ext2 zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate jbd sha512_generic sha256_generic sunrpc ipv6 microcode virtio_balloon virtio_net i 2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs] [25311.746280] [25311.746280] Pid: 25007, comm: ll_ost_io01_003 Tainted: P B --------------- 2.6.32-debug #5 Bochs Bochs [25311.746280] RIP: 0010:[&lt;ffffffff8127e0db&gt;] [&lt;ffffffff8127e0db&gt;] memcpy+0xb/0x120 [25311.746280] RSP: 0018:ffff88009fe4d508 EFLAGS: 00010246 [25311.746280] RAX: ffff8800bd658000 RBX: 0000000000000000 RCX: 0000000000000200 [25311.746280] RDX: 0000000000000000 RSI: ffff88004bfae000 RDI: ffff8800bd658000 [25311.746280] RBP: ffff88009fe4d570 R08: 0000000000001000 R09: ffff8800560c8a20 [25311.746280] R10: 0000000000000000 R11: ffff8800bd658000 R12: 000000000003f000 [25311.746280] R13: 0000000000001000 R14: 0000000000000000 R15: ffff88004bfae000 [25311.746280] FS: 00007f9a3b65d700(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 [25311.746280] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [25311.746280] CR2: ffff8800bd658000 CR3: 0000000001a25000 CR4: 00000000000006e0 [25311.746280] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [25311.746280] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [25311.746280] Process ll_ost_io01_003 (pid: 25007, threadinfo ffff88009fe4c000, task ffff8800bd838100) [25311.746280] Stack: [25311.746280] ffffffffa07ba0f3 0000000000001000 ffff8800bd658000 ffff880108b89a20 [25311.746280] &lt;d&gt; ffff8800560c8a20 ffffea000296e340 0000004000000040 ffff88009fe4d580 [25311.746280] &lt;d&gt; ffff8800dd5a1df0 ffff880282e7edf0 ffff88018fff0df0 ffff88018fff0df0 [25311.746280] Call Trace: [25311.746280] [&lt;ffffffffa07ba0f3&gt;] ? lnet_copy_kiov2kiov+0x153/0x450 [lnet] [25311.746280] [&lt;ffffffffa07bea72&gt;] lolnd_recv+0xd2/0xe0 [lnet] [25311.746280] [&lt;ffffffffa07b70db&gt;] lnet_ni_recv+0xbb/0x380 [lnet] [25311.746280] [&lt;ffffffffa07be378&gt;] lnet_parse+0x1838/0x1900 [lnet] [25311.746280] [&lt;ffffffffa07beaab&gt;] lolnd_send+0x2b/0xb0 [lnet] [25311.746280] [&lt;ffffffffa07b6f5b&gt;] lnet_ni_send+0x4b/0x110 [lnet] [25311.746280] [&lt;ffffffffa07bb6d1&gt;] lnet_send+0x851/0xc10 [lnet] [25311.746280] [&lt;ffffffffa07bd9cd&gt;] lnet_parse+0xe8d/0x1900 [lnet] [25311.746280] [&lt;ffffffffa072c34f&gt;] ? cfs_trace_unlock_tcd+0x3f/0xa0 [libcfs] [25311.746280] [&lt;ffffffffa07beaab&gt;] lolnd_send+0x2b/0xb0 [lnet] [25311.746280] [&lt;ffffffffa07b6f5b&gt;] lnet_ni_send+0x4b/0x110 [lnet] [25311.746280] [&lt;ffffffffa07bb6d1&gt;] lnet_send+0x851/0xc10 [lnet] [25311.746280] [&lt;ffffffffa07bbd74&gt;] LNetGet+0x2e4/0x830 [lnet] [25311.746280] [&lt;ffffffffa0e44370&gt;] ptlrpc_start_bulk_transfer+0x160/0x650 [ptlrpc] [25311.746280] [&lt;ffffffffa0e15830&gt;] target_bulk_io+0x180/0x930 [ptlrpc] [25311.746280] [&lt;ffffffffa073d6d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [25311.746280] [&lt;ffffffffa072d8b5&gt;] ? cfs_waitq_init+0x15/0x20 [libcfs] [25311.746280] [&lt;ffffffffa0e3af5e&gt;] ? new_bulk+0x10e/0x220 [ptlrpc] [25311.746280] [&lt;ffffffffa0ba0db1&gt;] ? class_export_get+0x81/0x90 [obdclass] [25311.746280] [&lt;ffffffffa0e37ce8&gt;] ? __ptlrpc_prep_bulk_page+0x68/0x1a0 [ptlrpc] [25311.746280] [&lt;ffffffffa083045f&gt;] ost_brw_write+0x132f/0x15d0 [ost] [25311.746280] [&lt;ffffffffa073d6d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [25311.746280] [&lt;ffffffffa0835250&gt;] ost_handle+0x3120/0x4550 [ost] [25311.746280] [&lt;ffffffffa0739464&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [25311.746280] [&lt;ffffffffa0e58483&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [25311.746280] [&lt;ffffffffa072d66e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [25311.746280] [&lt;ffffffffa0e51171&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [25311.746280] [&lt;ffffffffa073d6d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [25311.746280] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [25311.746280] [&lt;ffffffffa0e5b01a&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [25311.746280] [&lt;ffffffffa0e5a480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [25311.746280] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [25311.746280] [&lt;ffffffffa0e5a480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [25311.746280] [&lt;ffffffffa0e5a480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [25311.746280] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [25311.746280] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 &lt;f3&gt; 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c [25311.746280] RIP [&lt;ffffffff8127e0db&gt;] memcpy+0xb/0x120 [25311.746280] RSP &lt;ffff88009fe4d508&gt; [25311.746280] CR2: ffff8800bd658000 [25311.746280] ---[ end trace 96e412e2254363a4 ]--- </pre> </div></div> <p>I now hit this very reliably in sanity test 224b with no zfs in picture.</p> <p>new trace looks like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;[11515.961133] Lustre: lustre-OST0001: Client 6c55d310-d816-6361-9358-41f6556f3804 (at 0@lo) reconnecting &lt;3&gt;[11516.960148] LustreError: 7151:0:(ldlm_lib.c:2701:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800b676abe8 x1467459440236164/t0(0) o3-&gt;6c55d310-d816-6361-9358-41f6556f3804@0@lo:0/0 lens 488/432 e 1 to 0 dl 1399478720 ref 1 fl Interpret:/0/0 rc 0/0 &lt;1&gt;[11518.960326] BUG: unable to handle kernel paging request at ffff880080b35000 &lt;1&gt;[11518.960635] IP: [&lt;ffffffff8128c91b&gt;] memcpy+0xb/0x120 &lt;4&gt;[11518.961004] PGD 1a26063 PUD 501067 PMD 507067 PTE 8000000080b35060 &lt;4&gt;[11518.961313] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC &lt;4&gt;[11518.961580] last sysfs file: /sys/devices/system/cpu/possible &lt;4&gt;[11518.961855] CPU 2 &lt;4&gt;[11518.961893] Modules linked in: lustre ofd osp lod ost mdt mdd mgs nodemap osd_ldiskfs ldiskfs lquota lfsck mgc lov osc mdc lmv fid fld ptlrpc obdclass ksocklnd lnet libcfs ext2 exportfs jbd sha512_generic sha256_generic ext4 jbd2 mbcache ppdev parport_pc parport virtio_balloon virtio_console i2c_piix4 i2c_core virtio_net virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod nfs lockd fscache auth_rpcgss nfs_acl sunrpc be2iscsi bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: obdecho] &lt;4&gt;[11518.964196] &lt;4&gt;[11518.964196] Pid: 11253, comm: ll_ost_io01_006 Tainted: G W --------------- 2.6.32-rhe6.5-debug #2 Bochs Bochs &lt;4&gt;[11518.964196] RIP: 0010:[&lt;ffffffff8128c91b&gt;] [&lt;ffffffff8128c91b&gt;] memcpy+0xb/0x120 &lt;4&gt;[11518.964196] RSP: 0018:ffff88003e7b5738 EFLAGS: 00010246 &lt;4&gt;[11518.964196] RAX: ffff8800aa7b2000 RBX: 0000000000000000 RCX: 0000000000000200 &lt;4&gt;[11518.964196] RDX: 0000000000000000 RSI: ffff880080b35000 RDI: ffff8800aa7b2000 &lt;4&gt;[11518.964196] RBP: ffff88003e7b57a0 R08: 0000000000001000 R09: ffff8800b7677370 &lt;4&gt;[11518.964196] R10: 0000000000000000 R11: ffff8800aa7b2000 R12: 0000000000000000 &lt;4&gt;[11518.964196] R13: 0000000000001000 R14: 0000000000000000 R15: ffff880080b35000 &lt;4&gt;[11518.964196] FS: 0000000000000000(0000) GS:ffff880006300000(0000) knlGS:0000000000000000 &lt;4&gt;[11518.964196] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b &lt;4&gt;[11518.964196] CR2: ffff880080b35000 CR3: 0000000001a25000 CR4: 00000000000006e0 &lt;4&gt;[11518.964196] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;[11518.964196] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;[11518.964196] Process ll_ost_io01_006 (pid: 11253, threadinfo ffff88003e7b4000, task ffff8800a92fa500) &lt;4&gt;[11518.964196] Stack: &lt;4&gt;[11518.964196] ffffffffa0be7963 0000000000001000 ffff8800aa7b2000 ffff88009f149b68 &lt;4&gt;[11518.964196] &lt;d&gt; ffff8800b7677370 ffffea000254aef0 0000000100000001 0000000000000000 &lt;4&gt;[11518.964196] &lt;d&gt; ffff8800552a1df0 ffff880057507df0 ffff8800268f3df0 ffff8800268f3df0 &lt;4&gt;[11518.964196] Call Trace: &lt;4&gt;[11518.964196] [&lt;ffffffffa0be7963&gt;] ? lnet_copy_kiov2kiov+0x153/0x410 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0bec472&gt;] lolnd_recv+0xd2/0xe0 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffff8151677e&gt;] ? _spin_unlock+0xe/0x10 &lt;4&gt;[11518.964196] [&lt;ffffffffa0be4f7b&gt;] lnet_ni_recv+0xbb/0x320 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0be5263&gt;] lnet_recv_put+0x83/0xb0 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0beb7f4&gt;] lnet_parse+0x1254/0x18c0 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0bec4ab&gt;] lolnd_send+0x2b/0xa0 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0be4e1b&gt;] lnet_ni_send+0x4b/0xf0 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0be9200&gt;] lnet_send+0x850/0xb70 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0bea05a&gt;] LNetPut+0x31a/0x860 [lnet] &lt;4&gt;[11518.964196] [&lt;ffffffffa0701487&gt;] ptlrpc_start_bulk_transfer+0x1e7/0x720 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 &lt;4&gt;[11518.964196] [&lt;ffffffff8108292b&gt;] ? try_to_del_timer_sync+0x7b/0xe0 &lt;4&gt;[11518.964196] [&lt;ffffffffa06d0318&gt;] target_bulk_io+0x6e8/0xa00 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffff81514239&gt;] ? schedule_timeout+0x199/0x2e0 &lt;4&gt;[11518.964196] [&lt;ffffffffa076a3e5&gt;] tgt_brw_read+0x10e5/0x1150 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffffa0706cad&gt;] ? lustre_pack_reply_v2+0x1fd/0x2a0 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt;[11518.964196] [&lt;ffffffffa0706dfe&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffffa076754c&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffffa0717f88&gt;] ptlrpc_main+0xcc8/0x1950 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffffa07172c0&gt;] ? ptlrpc_main+0x0/0x1950 [ptlrpc] &lt;4&gt;[11518.964196] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 &lt;4&gt;[11518.964196] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 &lt;4&gt;[11518.964196] [&lt;ffffffff81098b70&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[11518.964196] [&lt;ffffffff8100c240&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[11518.964196] Code: 49 89 70 50 19 c0 49 89 70 58 41 c6 40 4c 04 83 e0 fc 83 c0 08 41 88 40 4d c9 c3 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 &lt;f3&gt; 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c &lt;1&gt;[11518.964196] RIP [&lt;ffffffff8128c91b&gt;] memcpy+0xb/0x120 &lt;4&gt;[11518.964196] RSP &lt;ffff88003e7b5738&gt; &lt;4&gt;[11518.964196] CR2: ffff880080b35000 </pre> </div></div> <p>Just had this happen again on a new setup, current master.<br/> Test 224b again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10301.251331] BUG: unable to handle kernel paging request at ffff88029aa23000 [10301.252837] IP: [&lt;ffffffff8138590d&gt;] memcpy+0xd/0x110 [10301.253477] PGD 2e75067 PUD 33ebfa067 PMD 33eb24067 PTE 800000029aa23060 [10301.254185] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [10301.254823] Modules linked in: brd lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate mbcache jbd2 syscopyarea sysfillrect sysimgblt ttm drm_kms_helper ata_generic pata_acpi i2c_piix4 drm virtio_blk floppy pcspkr ata_piix serio_raw virtio_console virtio_balloon i2c_core libata nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [10301.260919] CPU: 5 PID: 4882 Comm: ll_ost_io00_002 Tainted: P OE ------------ 3.10.0-debug #2 [10301.262166] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [10301.262860] task: ffff8801aa070440 ti: ffff8802cc144000 task.ti: ffff8802cc144000 [10301.264050] RIP: 0010:[&lt;ffffffff8138590d&gt;] [&lt;ffffffff8138590d&gt;] memcpy+0xd/0x110 [10301.265262] RSP: 0018:ffff8802cc147630 EFLAGS: 00010246 [10301.265898] RAX: ffff8802f8d55000 RBX: 0000000000001000 RCX: 0000000000000200 [10301.266563] RDX: 0000000000000000 RSI: ffff88029aa23000 RDI: ffff8802f8d55000 [10301.267255] RBP: ffff8802cc147688 R08: ffff8802f8d55000 R09: ffff88029aa23000 [10301.268146] R10: ffff88024abd5fe0 R11: ffff8802cf93ffe0 R12: 0000000000000000 [10301.269011] R13: 0000000000000000 R14: ffff8802f8d55000 R15: 0000000000000000 [10301.269878] FS: 0000000000000000(0000) GS:ffff88033e4a0000(0000) knlGS:0000000000000000 [10301.271249] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [10301.271681] CR2: ffff88029aa23000 CR3: 0000000001c0e000 CR4: 00000000000006e0 [10301.273318] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [10301.273984] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [10301.274608] Stack: [10301.275164] ffffffffa03035d1 0000000100000001 ffff8802cf93ffe0 ffff88024abd5fe0 [10301.276335] 0000000000001000 ffff88029aa23000 ffff88023ac08e00 ffff880048b10e00 [10301.277496] 0000000000000000 ffff880048b10e00 ffff880048b10e00 ffff8802cc1476b0 [10301.279048] Call Trace: [10301.279640] [&lt;ffffffffa03035d1&gt;] ? lnet_copy_kiov2kiov+0x161/0x3e0 [lnet] [10301.280324] [&lt;ffffffffa030ac71&gt;] lolnd_recv+0xb1/0xc0 [lnet] [10301.280976] [&lt;ffffffffa0305fb6&gt;] lnet_ni_recv+0xc6/0x320 [lnet] [10301.282541] [&lt;ffffffffa03066f1&gt;] lnet_recv_put+0x81/0xb0 [lnet] [10301.283254] [&lt;ffffffffa03088f6&gt;] lnet_parse_local+0x5a6/0xd40 [lnet] [10301.284530] [&lt;ffffffffa021abd8&gt;] ? cfs_percpt_lock+0x58/0x110 [libcfs] [10301.285273] [&lt;ffffffffa030993a&gt;] lnet_parse+0x8aa/0xfa0 [lnet] [10301.285936] [&lt;ffffffffa030ae0b&gt;] lolnd_send+0x2b/0xa0 [lnet] [10301.286611] [&lt;ffffffffa0301e0e&gt;] lnet_ni_send+0x3e/0xd0 [lnet] [10301.287301] [&lt;ffffffffa03071f7&gt;] lnet_send+0x77/0x180 [lnet] [10301.287962] [&lt;ffffffffa0307545&gt;] LNetPut+0x245/0x7a0 [lnet] [10301.288695] [&lt;ffffffffa09fc476&gt;] ptlrpc_start_bulk_transfer+0x2d6/0x790 [ptlrpc] [10301.289935] [&lt;ffffffff8139138b&gt;] ? free_object+0x8b/0xb0 [10301.290594] [&lt;ffffffff8139138b&gt;] ? free_object+0x8b/0xb0 [10301.291301] [&lt;ffffffffa09c0173&gt;] target_bulk_io+0x823/0xb20 [ptlrpc] [10301.292021] [&lt;ffffffffa0a686e3&gt;] tgt_brw_read+0x773/0x1870 [ptlrpc] [10301.292726] [&lt;ffffffff811cd4f9&gt;] ? __kmalloc+0x649/0x660 [10301.293920] [&lt;ffffffff817063d7&gt;] ? _raw_spin_unlock+0x27/0x40 [10301.294613] [&lt;ffffffff810f3b93&gt;] ? is_module_address+0x23/0x30 [10301.295443] [&lt;ffffffff810e24ac&gt;] ? static_obj+0x3c/0x50 [10301.304053] [&lt;ffffffff810b7cc0&gt;] ? wake_up_state+0x20/0x20 [10301.304763] [&lt;ffffffffa0a65ccb&gt;] tgt_request_handle+0x93b/0x1390 [ptlrpc] [10301.305470] [&lt;ffffffffa0a10351&gt;] ptlrpc_server_handle_request+0x251/0xae0 [ptlrpc] [10301.306690] [&lt;ffffffffa0a14108&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [10301.307383] [&lt;ffffffffa0a136b0&gt;] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc] [10301.308850] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [10301.309462] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 [10301.310126] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [10301.310746] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 [10301.311400] Code: 43 4e 5b 5d c3 66 0f 1f 84 00 00 00 00 00 e8 fb fb ff ff eb e2 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 &lt;f3&gt; 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c [10301.342280] RIP [&lt;ffffffff8138590d&gt;] memcpy+0xd/0x110 [10301.342945] RSP &lt;ffff8802cc147630&gt; [10301.343550] CR2: ffff88029aa23000 </pre> </div></div> Development Rank 1|hzva9z: Rank (Obsolete) 5200 Severity