[LU-7634] general protection fault in dio_complete_routine

Thu, 10 Mar 2016 22:01:50 +0000

Running obdfilter triggered GPF on 2 different servers. Duplicate of LU-6654 but in Lustre2.7.1?

<code>
[631929.395337] general protection fault: 0000 1 SMP ^M
[631929.410516] Modules linked in: obdecho(OF) osp(OF) ofd(OF) lfsck(OF) ost(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) ldiskfs(OF) dm_cache_mq dm_cache dm_persistent_data dm_bio_prison dm_bufio libcrc32c lustre(OF) lmv(OF) mdc(OF) lov(OF) fid(OF) fld(OF) ko2iblnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) sha512_generic libcfs(OF) bonding xprtrdma(OF) sunrpc iscsi_target_mod target_core_mod ib_iser(OF) libiscsi scsi_transport_iscsi ib_srpt(OF) ib_srp(OF) scsi_transport_srp(OF) ib_ipoib(OF) rdma_ucm(OF) ib_ucm(OF) ib_uverbs(OF) ib_umad(OF) rdma_cm(OF) ib_cm(OF) iw_cm(OF) dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support dm_round_robin intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel cryptd pcspkr dm_service_time sb_edac ipmi_devintf lpc_ich mei_me edac_core mfd_core wmi mei ioatdma i2c_i801 shpchp ipmi_si ipmi_msghandler acpi_pad acpi_power_meter dm_multipath dm_mod uinput binfmt_misc tcp_bic ext4 mbcache jbd2 mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) vxlan ip_tunnel sd_mod ast syscopyarea sysfillrect sysimgblt igb drm_kms_helper ahci lpfc ptp ttm libahci pps_core mlx4_core(OF) mpt3sas crc_t10dif crct10dif_common raid_class drm dca mlx_compat(OF) scsi_transport_fc i2c_algo_bit libata nvme scsi_transport_sas scsi_tgt i2c_core memtrack(OF)^M
[631929.759883] CPU: 12 PID: 25826 Comm: lctl Tainted: GF O-------------- 3.10.0-229.20.1.el7.20151203.x86_64.lustre271 #1^M
[631929.795042] Hardware name: SGI.COM CH-C2112-GP2-G/X10DRU-i+, BIOS 1.0b 05/08/2015^M
[631929.817713] task: ffff882025448000 ti: ffff881d9b114000 task.ti: ffff881d9b114000^M
[631929.840379] RIP: 0010:[<ffffffff810a097b>] [<ffffffff810a097b>] __wake_up_common+0x2b/0x90^M
[631929.865676] RSP: 0018:ffff88207fc03a90 EFLAGS: 00010086^M
[631929.881838] RAX: 0000000000000246 RBX: ffff881950763a80 RCX: 0000000000000000^M
[631929.903464] RDX: 5a5a5a5a5a5a5a5a RSI: 0000000000000003 RDI: ffff881950763a80^M
[631929.925092] RBP: ffff88207fc03ac8 R08: 0000000000000000 R09: 000000018066001c^M
[631929.946719] R10: ffffffff8115c207 R11: ffffea0036ea0800 R12: ffff881950763a88^M
[631929.968347] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003^M
[631929.989974] FS: 00007fffedae6740(0000) GS:ffff88207fc00000(0000) knlGS:0000000000000000^M
[631930.014463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M
[631930.031928] CR2: 000000000063bd98 CR3: 0000001dba0d0000 CR4: 00000000001407e0^M
[631930.053555] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000^M
[631930.075181] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400^M
[631930.096809] Stack:^M
[631930.103083] 0000000128bed360 0000000000000000 ffff881950763a80 0000000000000246^M
[631930.125593] 0000000000000003 0000000000000001 0000000000000000 ffff88207fc03b00^M
[631930.148104] ffffffff810a25c9 ffff881134be1000 ffff881027640000 ffff881134be1000^M
[631930.170617] Call Trace:^M
[631930.178192] <IRQ> ^M
[631930.184204] [<ffffffff810a25c9>] __wake_up+0x39/0x50^M
[631930.200175] [<ffffffffa106afb1>] dio_complete_routine+0xf1/0x210 [osd_ldiskfs]^M
[631930.222334] [<ffffffff8115c207>] ? mempool_free_slab+0x17/0x20^M
[631930.240315] [<ffffffff8115c609>] ? mempool_free+0x49/0x90^M
[631930.257003] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631930.272383] [<ffffffffa0274a7a>] dec_pending+0x18a/0x2e0 [dm_mod]^M
[631930.291147] [<ffffffffa0275748>] clone_endio+0xc8/0xf0 [dm_mod]^M
[631930.309386] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631930.324771] [<ffffffffa0274a7a>] dec_pending+0x18a/0x2e0 [dm_mod]^M
[631930.343536] [<ffffffffa0275748>] clone_endio+0xc8/0xf0 [dm_mod]^M
[631930.361775] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631930.377161] [<ffffffff812b0af0>] blk_update_request+0x90/0x350^M
[631930.395145] [<ffffffffa02750a8>] end_clone_bio+0x48/0x80 [dm_mod]^M
[631930.413906] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631930.429285] [<ffffffff812b0af0>] blk_update_request+0x90/0x350^M
[631930.447269] [<ffffffff812b0dcc>] blk_update_bidi_request+0x1c/0x80^M
[631930.466296] [<ffffffff812b10af>] blk_end_bidi_request+0x1f/0x60^M
[631930.484538] [<ffffffff812b1100>] blk_end_request+0x10/0x20^M
[631930.501483] [<ffffffff813fd558>] scsi_io_completion+0x108/0x650^M
[631930.519726] [<ffffffff813f24d3>] scsi_finish_command+0xb3/0x110^M
[631930.537970] [<ffffffff813fd35f>] scsi_softirq_done+0x12f/0x160^M
[631930.555955] [<ffffffff812b7630>] blk_done_softirq+0x90/0xc0^M
[631930.573157] [<ffffffff81077c07>] __do_softirq+0xf7/0x290^M
[631930.589582] [<ffffffff8161965c>] call_softirq+0x1c/0x30^M
[631930.605742] [<ffffffff81015de5>] do_softirq+0x55/0x90^M
[631930.621383] [<ffffffff81077fa5>] irq_exit+0x115/0x120^M
[631930.637024] [<ffffffff8161a1f8>] do_IRQ+0x58/0xf0^M
[631930.651628] [<ffffffff8160f42d>] common_interrupt+0x6d/0x6d^M
[631930.674837] [<ffffffffa099d44a>] ? lprocfs_counter_sub+0x2a/0x180 [obdclass]^M
[631930.697067] [<ffffffffa1257087>] echo_client_prep_commit.isra.50+0xca7/0xed0 [obdecho]^M
[631930.721297] [<ffffffffa126015e>] echo_client_iocontrol+0xb4e/0x2060 [obdecho]^M
[631930.743194] [<ffffffffa0987cbd>] class_handle_ioctl+0x1bad/0x2330 [obdclass]^M
[631930.764836] [<ffffffff810d4fd0>] ? futex_wake+0x80/0x160^M
[631930.781258] [<ffffffff8126c1e8>] ? security_capable+0x18/0x20^M
[631930.798987] [<ffffffffa096e5e2>] obd_class_ioctl+0xd2/0x170 [obdclass]^M
[631930.819045] [<ffffffff811dd095>] do_vfs_ioctl+0x2e5/0x4c0^M
[631930.835727] [<ffffffff811dd311>] SyS_ioctl+0xa1/0xc0^M
[631930.851108] [<ffffffff81617d49>] system_call_fastpath+0x16/0x1b^M
[631930.869350] Code: 0f 1f 44 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41 54 4c 8d 67 08 53 48 83 ec 10 89 55 cc 48 8b 57 08 4c 89 45 d0 <48> 8b 0a 49 39 d4 48 8d 42 e8 4c 8d 69 e8 75 0b eb 3b 0f 1f 00 ^M
[631930.927802] RIP [<ffffffff810a097b>] __wake_up_common+0x2b/0x90^M
[631930.946072] RSP <ffff88207fc03a90>^M
[631930.957103] -----------~~[ cut here ]~~-----------^M
[631930.971187] kernel BUG at mm/vmalloc.c:1339!^M
[631930.984227] invalid opcode: 0000 2 SMP ^M
[631930.996796] Modules linked in: obdecho(OF) osp(OF) ofd(OF) lfsck(OF) ost(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) ldiskfs(OF) dm_cache_mq dm_cache dm_persistent_data dm_bio_prison dm_bufio libcrc32c lustre(OF) lmv(OF) mdc(OF) lov(OF) fid(OF) fld(OF) ko2iblnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) sha512_generic libcfs(OF) bonding xprtrdma(OF) sunrpc iscsi_target_mod target_core_mod ib_iser(OF) libiscsi scsi_transport_iscsi ib_srpt(OF) ib_srp(OF) scsi_transport_srp(OF) ib_ipoib(OF) rdma_ucm(OF) ib_ucm(OF) ib_uverbs(OF) ib_umad(OF) rdma_cm(OF) ib_cm(OF) iw_cm(OF) dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support dm_round_robin intel_powerclamp coretemp intel_rapl kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel cryptd pcspkr dm_service_time sb_edac ipmi_devintf lpc_ich mei_me edac_core mfd_core wmi mei ioatdma i2c_i801 shpchp ipmi_si ipmi_msghandler acpi_pad acpi_power_meter dm_multipath dm_mod uinput binfmt_misc tcp_bic ext4 mbcache jbd2 mlx4_ib(OF) ib_sa(OF) ib_mad(OF) ib_core(OF) ib_addr(OF) vxlan ip_tunnel sd_mod ast syscopyarea sysfillrect sysimgblt igb drm_kms_helper ahci lpfc ptp ttm libahci pps_core mlx4_core(OF) mpt3sas crc_t10dif crct10dif_common raid_class drm dca mlx_compat(OF) scsi_transport_fc i2c_algo_bit libata nvme scsi_transport_sas scsi_tgt i2c_core memtrack(OF)^M
[631931.346165] CPU: 12 PID: 25826 Comm: lctl Tainted: GF O-------------- 3.10.0-229.20.1.el7.20151203.x86_64.lustre271 #1^M
[631931.381324] Hardware name: SGI.COM CH-C2112-GP2-G/X10DRU-i+, BIOS 1.0b 05/08/2015^M
[631931.403992] task: ffff882025448000 ti: ffff881d9b114000 task.ti: ffff881d9b114000^M
[631931.426661] RIP: 0010:[<ffffffff811930ae>] [<ffffffff811930ae>] __get_vm_area_node+0x1ce/0x1d0^M
[631931.453000] RSP: 0018:ffff88207fc03150 EFLAGS: 00010006^M
[631931.469159] RAX: ffff881d9b117fd8 RBX: 00000000ffffffff RCX: ffffc90000000000^M
[631931.490789] RDX: 0000000000000022 RSI: 0000000000000001 RDI: 0000000000002000^M
[631931.512414] RBP: ffff88207fc031b0 R08: ffffe8ffffffffff R09: 00000000ffffffff^M
[631931.534041] R10: ffffffffa0062f44 R11: ffff8810262c38b8 R12: ffffffffa00992c9^M
[631931.555669] R13: 0000000000001800 R14: 00000000000080d2 R15: ffffea0000de1880^M
[631931.577297] FS: 00007fffedae6740(0000) GS:ffff88207fc00000(0000) knlGS:0000000000000000^M
[631931.601785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M
[631931.619249] CR2: 000000000063bd98 CR3: 0000001dba0d0000 CR4: 00000000001407e0^M
[631931.640876] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000^M
[631931.662504] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400^M
[631931.684130] Stack:^M
[631931.690402] ffffffff8119481d 00000000000080d2 ffffffffa00992c9 8000000000000163^M
[631931.712915] 000080d200000000 0000000000000000 00000000ee351738 ffff881c2466fae0^M
[631931.735427] ffff8810262c3898 0000000000300000 0000000000000080 ffffea0000de1880^M
[631931.757938] Call Trace:^M
[631931.765512] <IRQ> ^M
[631931.771524] [<ffffffff8119481d>] ? __vmalloc_node_range+0x7d/0x270^M
[631931.791127] [<ffffffffa00992c9>] ? ttm_tt_init+0x69/0xb0 [ttm]^M
[631931.809108] [<ffffffff81194a51>] __vmalloc+0x41/0x50^M
[631931.824491] [<ffffffffa00992c9>] ? ttm_tt_init+0x69/0xb0 [ttm]^M
[631931.842474] [<ffffffffa00992c9>] ttm_tt_init+0x69/0xb0 [ttm]^M
[631931.859938] [<ffffffffa0062f68>] ast_ttm_tt_create+0x58/0x90 [ast]^M
[631931.878964] [<ffffffffa0099a7d>] ttm_bo_add_ttm+0x8d/0xc0 [ttm]^M
[631931.897208] [<ffffffffa009b0f1>] ttm_bo_handle_move_mem+0x571/0x5b0 [ttm]^M
[631931.918052] [<ffffffff81604654>] ? __slab_free+0x10e/0x277^M
[631931.934996] [<ffffffffa009b74a>] ? ttm_bo_mem_space+0x10a/0x310 [ttm]^M
[631931.954801] [<ffffffffa009be17>] ttm_bo_validate+0x247/0x260 [ttm]^M
[631931.973825] [<ffffffff81059e69>] ? iounmap+0x79/0xa0^M
[631931.989210] [<ffffffff81050069>] ? kgdb_arch_late+0xe9/0x180^M
[631932.006670] [<ffffffffa0063592>] ast_bo_push_sysram+0x82/0xe0 [ast]^M
[631932.025954] [<ffffffffa0060fb4>] ast_crtc_do_set_base.isra.13.constprop.23+0x84/0x370 [ast]^M
[631932.051484] [<ffffffffa005f2ca>] ? ast_set_index_reg_mask+0x6a/0x80 [ast]^M
[631932.072332] [<ffffffffa0061d94>] ast_crtc_mode_set+0xaf4/0xc50 [ast]^M
[631932.091882] [<ffffffffa08c7939>] drm_crtc_helper_set_mode+0x2e9/0x520 [drm_kms_helper]^M
[631932.116109] [<ffffffffa08c86bf>] drm_crtc_helper_set_config+0x87f/0xaa0 [drm_kms_helper]^M
[631932.140856] [<ffffffff8160be5b>] ? __ww_mutex_lock+0x1b/0xa0^M
[631932.158331] [<ffffffffa01ac711>] drm_mode_set_config_internal+0x61/0xe0 [drm]^M
[631932.180236] [<ffffffffa08d0a94>] drm_fb_helper_pan_display+0x94/0xf0 [drm_kms_helper]^M
[631932.204204] [<ffffffff81328ea9>] fb_pan_display+0xc9/0x190^M
[631932.221144] [<ffffffff81337f30>] bit_update_start+0x20/0x50^M
[631932.238345] [<ffffffff8133795d>] fbcon_switch+0x39d/0x5a0^M
[631932.255032] [<ffffffff813a61a9>] redraw_screen+0x1a9/0x270^M
[631932.271971] [<ffffffff813290ae>] ? fb_blank+0xae/0xc0^M
[631932.287612] [<ffffffff81334e7a>] fbcon_blank+0x22a/0x2f0^M
[631932.304037] [<ffffffff81070394>] ? wake_up_klogd+0x34/0x50^M
[631932.320978] [<ffffffff810705b8>] ? console_unlock+0x208/0x400^M
[631932.338704] [<ffffffff8107ee73>] ? __internal_add_timer+0x113/0x130^M
[631932.357985] [<ffffffff8107f067>] ? internal_add_timer+0x17/0x40^M
[631932.376229] [<ffffffff81080c0d>] ? mod_timer+0x11d/0x240^M
[631932.392650] [<ffffffff813a6868>] do_unblank_screen+0xb8/0x1f0^M
[631932.410374] [<ffffffff813a69b0>] unblank_screen+0x10/0x20^M
[631932.427058] [<ffffffff812e77a9>] bust_spinlocks+0x19/0x40^M
[631932.443741] [<ffffffff816103e8>] oops_end+0x38/0x150^M
[631932.459121] [<ffffffff810173eb>] die+0x4b/0x70^M
[631932.472940] [<ffffffff8160fdce>] do_general_protection+0x11e/0x1b0^M
[631932.491966] [<ffffffff8160f6e8>] general_protection+0x28/0x30^M
[631932.509687] [<ffffffff8115c207>] ? mempool_free_slab+0x17/0x20^M
[631932.527672] [<ffffffff810a097b>] ? __wake_up_common+0x2b/0x90^M
[631932.545396] [<ffffffff810a25c9>] __wake_up+0x39/0x50^M
[631932.560784] [<ffffffffa106afb1>] dio_complete_routine+0xf1/0x210 [osd_ldiskfs]^M
[631932.582922] [<ffffffff8115c207>] ? mempool_free_slab+0x17/0x20^M
[631932.600907] [<ffffffff8115c609>] ? mempool_free+0x49/0x90^M
[631932.617590] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631932.632973] [<ffffffffa0274a7a>] dec_pending+0x18a/0x2e0 [dm_mod]^M
[631932.651737] [<ffffffffa0275748>] clone_endio+0xc8/0xf0 [dm_mod]^M
[631932.669979] [<ffffffff8120052d>] bio_endio+0x1d/0x30^M
[631932.789879] [<ffffffff812b0af0>] blk_update_request+0x90/0x350^M
[631932.807861] [<ffffffff812b0dcc>] blk_update_bidi_request+0x1c/0x80^M
[631932.826888] [<ffffffff812b10af>] blk_end_bidi_request+0x1f/0x60^M
[631932.845131] [<ffffffff812b1100>] blk_end_request+0x10/0x20^M
[631932.862074] [<ffffffff813fd558>] scsi_io_completion+0x108/0x650^M
[631932.880317] [<ffffffff813f24d3>] scsi_finish_command+0xb3/0x110^M
[631932.898563] [<ffffffff813fd35f>] scsi_softirq_done+0x12f/0x160^M
[631932.916544] [<ffffffff812b7630>] blk_done_softirq+0x90/0xc0^M
[631932.933748] [<ffffffff81077c07>] __do_softirq+0xf7/0x290^M
[631932.950169] [<ffffffff8161965c>] call_softirq+0x1c/0x30^M
[631932.966332] [<ffffffff81015de5>] do_softirq+0x55/0x90^M
[631932.981973] [<ffffffff81077fa5>] irq_exit+0x115/0x120^M
[631932.997614] [<ffffffff8161a1f8>] do_IRQ+0x58/0xf0^M
[631933.012215] [<ffffffff8160f42d>] common_interrupt+0x6d/0x6d^M
[631933.029416] <EOI> ^M
[631933.035427] [<ffffffffa099d44a>] ? lprocfs_counter_sub+0x2a/0x180 [obdclass]^M
[631933.057656] [<ffffffffa1257087>] echo_client_prep_commit.isra.50+0xca7/0xed0 [obdecho]^M
[631933.081888] [<ffffffffa126015e>] echo_client_iocontrol+0xb4e/0x2060 [obdecho]^M
[631933.103787] [<ffffffffa0987cbd>] class_handle_ioctl+0x1bad/0x2330 [obdclass]^M
[631933.125427] [<ffffffff810d4fd0>] ? futex_wake+0x80/0x160^M
[631933.141849] [<ffffffff8126c1e8>] ? security_capable+0x18/0x20^M
[631933.159579] [<ffffffffa096e5e2>] obd_class_ioctl+0xd2/0x170 [obdclass]^M
[631933.179636] [<ffffffff811dd095>] do_vfs_ioctl+0x2e5/0x4c0^M
[631933.196320] [<ffffffff811dd311>] SyS_ioctl+0xa1/0xc0^M
[631933.211701] [<ffffffff81617d49>] system_call_fastpath+0x16/0x1b^M
[631933.229942] Code: 4d 89 7c 24 50 e8 c3 be 47 00 48 83 c4 10 4c 89 f8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 4c 89 ff e8 49 ad 01 00 31 c0 e9 49 ff ff ff <0f> 0b 0f 1f 44 00 00 55 49 89 c8 41 b9 ff ff ff ff 48 89 d1 48 ^M
[631933.288395] RIP [<ffffffff811930ae>] __get_vm_area_node+0x1ce/0x1d0^M
[631933.307706] RSP <ffff88207fc03150>^M
[631933.318405] --~~[ end trace e73a23bb26c43759 ]~~--^M
[631933.440338] Kernel panic - not syncing: Fatal exception in interrupt^M
[631933.562409] drm_kms_helper: panic occurred, switching back to text console^M
[631933.583279] -----------~~[ cut here ]~~-----------^M
<code>

Whamcloud Community JIRA

[LU-7634] general protection fault in dio_complete_routine