[LU-10544] racer use after free in adler32_update Created: 21/Jan/18 Updated: 08/May/23 |
|
| Status: | Open |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor |
| Reporter: | Oleg Drokin | Assignee: | WC Triage |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||
| Severity: | 3 | ||||
| Rank (Obsolete): | 9223372036854775807 | ||||
| Description |
|
Just had this on current master out of the blue: [476681.006163] BUG: unable to handle kernel paging request at ffff8802f06c4000 [476681.007760] IP: [<ffffffffa020a170>] adler32_update+0x70/0x250 [libcfs] [476681.008785] PGD 2e75067 PUD 33e9f9067 PMD 33e875067 PTE 80000002f06c4060 [476681.009522] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [476681.010175] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea sysfillrect sysimgblt ttm drm_kms_helper ata_generic drm pata_acpi i2c_piix4 pcspkr ata_piix virtio_balloon serio_raw i2c_core virtio_console libata virtio_blk floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [476681.018904] CPU: 12 PID: 2137 Comm: ll_ost_io06_002 Tainted: P OE ------------ 3.10.0-debug #2 [476681.020320] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [476681.021084] task: ffff88029d2949c0 ti: ffff8802f1450000 task.ti: ffff8802f1450000 [476681.022819] RIP: 0010:[<ffffffffa020a170>] [<ffffffffa020a170>] adler32_update+0x70/0x250 [libcfs] [476681.024117] RSP: 0018:ffff8802f1453888 EFLAGS: 00010212 [476681.024773] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffffea000bc1b100 [476681.026257] RDX: 0000000000001000 RSI: ffff8802f06c4000 RDI: ffff8802f06c4000 [476681.027581] RBP: ffff8802f14538f0 R08: 0000000000000001 R09: 0000000000001000 [476681.029058] R10: 0000000000000000 R11: 0000000000000f40 R12: ffffea000bc1b102 [476681.030630] R13: 0000000000001000 R14: ffffffffa022c3d0 R15: 0000000000000001 [476681.032156] FS: 0000000000000000(0000) GS:ffff88033e580000(0000) knlGS:0000000000000000 [476681.041864] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [476681.042557] CR2: ffff8802f06c4000 CR3: 00000000ba661000 CR4: 00000000000006e0 [476681.043811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [476681.046746] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [476681.048311] Stack: [476681.049038] ffff8802d7789fd0 0000100000000000 ffff8802f06c4000 0000000000000000 [476681.051290] 0000100000000000 0000000000000000 000000006cf71350 ffff88033181cfc0 [476681.052826] ffff8802f06c4000 ffffea000bc1b102 0000000000001000 ffffffffa022c3d0 [476681.064622] Call Trace: [476681.065266] [<ffffffff81333237>] crypto_shash_update+0x47/0x120 [476681.066002] [<ffffffff8133355e>] shash_ahash_update+0x3e/0x70 [476681.066775] [<ffffffff813335a2>] shash_async_update+0x12/0x20 [476681.067544] [<ffffffffa0209143>] cfs_crypto_hash_update_page+0x93/0xc0 [libcfs] [476681.069023] [<ffffffffa0682ae6>] tgt_checksum_niobuf.isra.37+0x286/0x600 [ptlrpc] [476681.073433] [<ffffffffa068725f>] tgt_brw_read+0xc8f/0x1fa0 [ptlrpc] [476681.074195] [<ffffffff811cd4f9>] ? __kmalloc+0x649/0x660 [476681.074931] [<ffffffff817063d7>] ? _raw_spin_unlock+0x27/0x40 [476681.078624] [<ffffffff810e3201>] ? lockdep_init_map+0xa1/0x600 [476681.079394] [<ffffffffa06233f7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [476681.080816] [<ffffffffa062376c>] ? lustre_pack_reply_v2+0x16c/0x2a0 [ptlrpc] [476681.082222] [<ffffffffa0623912>] ? lustre_pack_reply_flags+0x72/0x1f0 [ptlrpc] [476681.088457] [<ffffffffa0623aa1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [476681.090318] [<ffffffffa0689c8b>] tgt_request_handle+0x93b/0x13e0 [ptlrpc] [476681.091110] [<ffffffffa062ec21>] ptlrpc_server_handle_request+0x261/0xaf0 [ptlrpc] [476681.092523] [<ffffffffa06329d8>] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [476681.093298] [<ffffffffa0631f80>] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc] [476681.095017] [<ffffffff810a2eba>] kthread+0xea/0xf0 [476681.095722] [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140 [476681.096469] [<ffffffff8170fb98>] ret_from_fork+0x58/0x90 [476681.097167] [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140 [476681.098303] Code: 44 00 00 8b 5d b8 b8 b0 15 00 00 81 fb b0 15 00 00 0f 46 c3 29 45 b8 83 f8 0f 89 45 a4 0f 8e f8 00 00 00 48 8b 7d a8 89 45 bc 90 <44> 0f b6 2f 44 0f b6 77 01 48 83 c7 10 44 0f b6 67 f2 0f b6 5f [476681.101124] RIP [<ffffffffa020a170>] adler32_update+0x70/0x250 [libcfs] [476681.101979] RSP <ffff8802f1453888> [476681.102719] CR2: ffff8802f06c4000 |
| Comments |
| Comment by Oleg Drokin [ 04/Feb/18 ] |
|
just had this once more in master-next |
| Comment by Oleg Drokin [ 07/Mar/18 ] |
|
and again |
| Comment by Oleg Drokin [ 27/Sep/19 ] |
|
So after a big gap, this issue seem to have returned in a different checksum handler [101837.120015] BUG: unable to handle kernel paging request at ffff8802d9f04000 [101837.120015] IP: [<ffffffffa00a93e1>] crc_array+0x0/0x1e [crc32c_intel] [101837.120015] PGD 241b067 PUD 33e9f9067 PMD 33e929067 PTE 80000002d9f04060 [101837.120015] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [101837.143463] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks ata_piix serio_raw crc32c_intel virtio_blk i2c_core libata floppy [last unloaded: libcfs] [101837.210942] CPU: 4 PID: 1787 Comm: ll_ost_io02_013 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.6-debug #1 [101837.268504] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [101837.275184] task: ffff8802ed752800 ti: ffff8802122f8000 task.ti: ffff8802122f8000 [101837.275184] RIP: 0010:[<ffffffffa00a93e1>] [<ffffffffa00a93e1>] crc_array+0x0/0x1e [crc32c_intel] [101837.275184] RSP: 0018:ffff8802122fb870 EFLAGS: 00010246 [101837.275184] RAX: 0000000000000080 RBX: 0000000000001000 RCX: ffff8802d9f04400 [101837.275184] RDX: ffff8802d9f04800 RSI: 0000000000001000 RDI: 0000000000000000 [101837.307009] RBP: ffff8802122fb8b0 R08: 00000000ffffffff R09: 0000000000000000 [101837.307009] R10: 0000000000000000 R11: ffff8802d9f04c00 R12: ffff880086f46290 [101837.307009] R13: ffff8802d9f04000 R14: ffff8802122f8000 R15: ffff880086f46290 [101837.307009] FS: 0000000000000000(0000) GS:ffff88033db00000(0000) knlGS:0000000000000000 [101837.307009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [101837.307009] CR2: ffff8802d9f04000 CR3: 0000000001c10000 CR4: 00000000001607e0 [101837.307009] Call Trace: [101837.307009] [<ffffffffa00a92eb>] ? crc32c_pcl_intel_update+0x7b/0xb0 [crc32c_intel] [101837.307009] [<ffffffff813910b7>] crypto_shash_update+0x47/0x120 [101837.307009] [<ffffffff813913de>] shash_ahash_update+0x3e/0x70 [101837.307009] [<ffffffff81391422>] shash_async_update+0x12/0x20 [101837.307009] [<ffffffffa02b2453>] cfs_crypto_hash_update_page+0x93/0xc0 [libcfs] [101837.307009] [<ffffffffa0760eae>] tgt_checksum_niobuf_rw+0x8ce/0xea0 [ptlrpc] [101837.307009] [<ffffffffa03fd775>] ? lprocfs_stats_unlock+0x45/0x50 [obdclass] [101837.307009] [<ffffffffa03ff7a9>] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [101837.307009] [<ffffffffa0723db1>] ? __req_capsule_get+0x161/0x710 [ptlrpc] [101837.307009] [<ffffffffa04336f0>] ? obd_dif_crc_fn+0x20/0x20 [obdclass] [101837.387516] [<ffffffffa0763d1d>] tgt_brw_read+0xc1d/0x1dd0 [ptlrpc] [101837.387516] [<ffffffffa03ff7a9>] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [101837.387516] [<ffffffffa0736d66>] ? null_alloc_rs+0x176/0x330 [ptlrpc] [101837.387516] [<ffffffffa06febcf>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [101837.387516] [<ffffffffa06fed51>] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [101837.387516] [<ffffffffa07678c5>] tgt_request_handle+0x985/0x1630 [ptlrpc] [101837.387516] [<ffffffffa031efae>] ? libcfs_nid2str_r+0xfe/0x130 [lnet] [101837.387516] [<ffffffffa070ad80>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc] [101837.387516] [<ffffffff810bfbd8>] ? __wake_up_common+0x58/0x90 [101837.469366] [<ffffffff813fb7bb>] ? do_raw_spin_unlock+0x4b/0x90 [101837.469366] [<ffffffffa070ef18>] ptlrpc_main+0xca8/0x1ca0 [ptlrpc] [101837.473426] [<ffffffff810c32ed>] ? finish_task_switch+0x5d/0x1b0 [101837.478137] [<ffffffffa070e270>] ? ptlrpc_register_service+0xff0/0xff0 [ptlrpc] [101837.478137] [<ffffffff810b4ed4>] kthread+0xe4/0xf0 [101837.478137] [<ffffffff810b4df0>] ? kthread_create_on_node+0x140/0x140 [101837.478137] [<ffffffff817c4c5d>] ret_from_fork_nospec_begin+0x7/0x21 [101837.478137] [<ffffffff810b4df0>] ? kthread_create_on_node+0x140/0x140 |