[LU-9942] Use after free in mdt_mfd_close->lu_object_put

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Minor
• Reporter: Oleg Drokin
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

Just had this hit on latest master-next in racer

[89073.094885] BUG: unable to handle kernel paging request at ffff8802f2350e48
[89073.096794] IP: [<ffffffffa03b9150>] lu_object_put+0x280/0x3d0 [obdclass]
[89073.097707] PGD 2e75067 PUD 33e9f9067 PMD 33e867067 PTE 80000002f2350060
[89073.098613] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[89073.099508] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) osc(OE) mdc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate mbcache jbd2 syscopyarea sysfillrect sysimgblt ttm ata_generic drm_kms_helper pata_acpi drm ata_piix i2c_piix4 virtio_console libata serio_raw pcspkr floppy virtio_blk i2c_core virtio_balloon nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs]
[89073.107406] CPU: 0 PID: 9198 Comm: mdt_rdpg00_001 Tainted: P           OE  ------------   3.10.0-debug #2
[89073.109003] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[89073.109606] task: ffff8802becf0700 ti: ffff8802f7e08000 task.ti: ffff8802f7e08000
[89073.111759] RIP: 0010:[<ffffffffa03b9150>]  [<ffffffffa03b9150>] lu_object_put+0x280/0x3d0 [obdclass]
[89073.128486] RSP: 0018:ffff8802f7e0bb88  EFLAGS: 00010246
[89073.129636] RAX: 0000000000000000 RBX: ffff8802f651e0d0 RCX: 0000000000000002
[89073.130498] RDX: 0000000000000002 RSI: ffffc900052c8000 RDI: ffff8802f2350e50
[89073.131103] RBP: ffff8802f7e0bbd8 R08: 0000000000000062 R09: 0000000000001d7e
[89073.131839] R10: 0000000000001a81 R11: 00000000003fffff R12: ffff8802c1967540
[89073.132528] R13: ffff8802f2350e88 R14: ffff8802f2350e38 R15: ffffc90005308048
[89073.133280] FS:  0000000000000000(0000) GS:ffff88033e400000(0000) knlGS:0000000000000000
[89073.134561] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[89073.135233] CR2: ffff8802f2350e48 CR3: 00000002baa90000 CR4: 00000000000006f0
[89073.135923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[89073.136633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[89073.137339] Stack:
[89073.137918]  ffffc90005308048 ffffc900052c8000 ffffc900052c8000 ffff880300003d03
[89073.139261]  00000000736a627b ffff8802c1967540 ffff8802c5195880 ffff880316d39800
[89073.140573]  ffff8802f2350e38 ffff8802eee54fa0 ffff8802f7e0bc28 ffffffffa0d0ce25
[89073.142922] Call Trace:
[89073.143561]  [<ffffffffa0d0ce25>] mdt_mfd_close+0x125/0x610 [mdt]
[89073.144815]  [<ffffffffa0d125dd>] mdt_close_internal+0xbd/0x220 [mdt]
[89073.145522]  [<ffffffffa0d12960>] mdt_close+0x220/0x720 [mdt]
[89073.146299]  [<ffffffffa0641783>] tgt_request_handle+0xa43/0x1330 [ptlrpc]
[89073.147037]  [<ffffffffa05eb8b1>] ptlrpc_server_handle_request+0x2a1/0xa70 [ptlrpc]
[89073.148343]  [<ffffffffa05ef588>] ptlrpc_main+0xa58/0x1de0 [ptlrpc]
[89073.149056]  [<ffffffff81706467>] ? _raw_spin_unlock_irq+0x27/0x50
[89073.149773]  [<ffffffffa05eeb30>] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc]
[89073.151028]  [<ffffffff810a2eba>] kthread+0xea/0xf0
[89073.151684]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[89073.152471]  [<ffffffff8170fb98>] ret_from_fork+0x58/0x90
[89073.153124]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[89073.153648] Code: ff 66 2e 0f 1f 84 00 00 00 00 00 48 8b 03 be 01 00 00 00 48 8b 7d c0 48 8b 40 40 ff 50 18 e9 4a fe ff ff 0f 1f 84 00 00 00 00 00 <49> 8b 46 10 a8 01 0f 84 36 fe ff ff 48 8b 7d b0 31 c9 31 d2 be 
[89073.155653] RIP  [<ffffffffa03b9150>] lu_object_put+0x280/0x3d0 [obdclass]

Issue Links

• is related to: LU-11204 mdt_reint_unlink->lu_object_put() crash

Comments

[Oleg Drokin]

This is still regularly happening in master:

[29297.568941] BUG: unable to handle kernel paging request at ffff880248775e60
[29297.569320] IP: [<ffffffffa0424d10>] lu_object_put+0x270/0x3c0 [obdclass]
[29297.569320] PGD 241b067 PUD 33edfb067 PMD 33edb7067 PTE 8000000248775060
[29297.628251] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[29297.628251] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm crct10dif_pclmul drm ata_piix crct10dif_common drm_panel_orientation_quirks crc32c_intel virtio_blk serio_raw libata i2c_core floppy [last unloaded: libcfs]
[29297.722659] 
[29297.722659] CPU: 7 PID: 21022 Comm: mdt_rdpg03_003 Kdump: loaded Tainted: P           OE  ------------   3.10.0-7.6-debug #1
[29297.722659] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[29297.722659] task: ffff8802f06905c0 ti: ffff880084fe0000 task.ti: ffff880084fe0000
[29297.722659] RIP: 0010:[<ffffffffa0424d10>]  [<ffffffffa0424d10>] lu_object_put+0x270/0x3c0 [obdclass]
[29297.722659] RSP: 0018:ffff880084fe3b80  EFLAGS: 00010246
[29297.722659] RAX: 0000000000000000 RBX: ffff880234dea160 RCX: 0000000000000002
[29297.722659] RDX: 0000000000000002 RSI: ffffc9000507f000 RDI: ffff880248775e68
[29297.722659] RBP: ffff880084fe3bd0 R08: 0000000000000039 R09: 000000000000036f
[29297.722659] R10: 0000000000002519 R11: 00000000003fffff R12: ffff880267187b00
[29297.722659] R13: ffff880248775ea0 R14: ffff880248775e50 R15: ffffc900050bf028
[29297.722659] FS:  0000000000000000(0000) GS:ffff88033dbc0000(0000) knlGS:0000000000000000
[29297.722659] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[29297.722659] CR2: ffff880248775e60 CR3: 00000002862fa000 CR4: 00000000001607e0
[29297.722659] Call Trace:
[29297.722659]  [<ffffffffa0db6bed>] mdt_mfd_close+0x21d/0x860 [mdt]
[29297.722659]  [<ffffffffa0dbc5f1>] mdt_close_internal+0xb1/0x220 [mdt]
[29297.722659]  [<ffffffffa0dbc980>] mdt_close+0x220/0x740 [mdt]
[29297.722659]  [<ffffffffa06c22f5>] tgt_request_handle+0x915/0x15c0 [ptlrpc]
[29297.722659]  [<ffffffffa029afa7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[29297.722659]  [<ffffffffa06652f9>] ptlrpc_server_handle_request+0x259/0xad0 [ptlrpc]
[29297.722659]  [<ffffffff810bfbd8>] ? __wake_up_common+0x58/0x90
[29297.722659]  [<ffffffff813fb7bb>] ? do_raw_spin_unlock+0x4b/0x90
[29297.722659]  [<ffffffffa06692ec>] ptlrpc_main+0xb5c/0x2040 [ptlrpc]
[29297.722659]  [<ffffffff810c32ed>] ? finish_task_switch+0x5d/0x1b0
[29297.722659]  [<ffffffffa0668790>] ? ptlrpc_register_service+0xfe0/0xfe0 [ptlrpc]
[29297.722659]  [<ffffffff810b4ed4>] kthread+0xe4/0xf0
[29297.722659]  [<ffffffff810b4df0>] ? kthread_create_on_node+0x140/0x140
[29297.722659]  [<ffffffff817c4c5d>] ret_from_fork_nospec_begin+0x7/0x21
[29297.722659]  [<ffffffff810b4df0>] ? kthread_create_on_node+0x140/0x140