-
Bug
-
Resolution: Fixed
-
Medium
-
Lustre 2.17.0, Lustre 2.18.0
-
None
-
3
-
9223372036854775807
For a while boilpot is plagued by crashes that manifest like this at module unload time:
Lustre: server umount lustre-OST0000 complete [98514.353936] Lustre: server umount lustre-MDT0000 complete [98517.533435] Key type lgssc unregistered [98517.874890] LNet: 3547322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [98517.900614] LNetError: 3547322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens3 not found [98518.977448] LNet: Removed LNI 192.168.123.91@tcp [98519.025122] general protection fault, probably for non-canonical address 0xdead000000000387: 0000 [#1] SMP DEBUG_PAGEALLOC [98519.028721] CPU: 10 PID: 3526148 Comm: kworker/10:0 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #1 [98519.030187] Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 [98519.031376] Workqueue: events cpu_latency_work [ptlrpc] [98519.033916] RIP: 0010:__dev_pm_qos_remove_request+0x16/0x210 [98519.034604] Code: 0a 00 03 01 eb d4 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 66 66 66 66 90 48 85 ff 0f 84 ec 01 00 00 41 56 41 55 41 54 55 53 <48> 8b 47 40 48 89 fb 48 85 c0 0f 84 5c 01 00 00 48 8b 90 90 02 00 [98519.038887] RSP: 0018:ffffb10bcbac7e28 EFLAGS: 00010286 [98519.041797] RAX: 0000000000000000 RBX: dead000000000387 RCX: 0000000633451944 [98519.042640] RDX: ffff8d1397c04d00 RSI: 0000000000000121 RDI: dead000000000387 [98519.049651] RBP: ffff8d13850ceaa0 R08: ffff8d139260104d R09: 0000000000000000 [98519.053309] R10: 00000002288de138 R11: ffff8d1392602000 R12: ffff8d13850ceaa8 [98519.055655] R13: ffff8d13850ceb70 R14: 0000000000000002 R15: ffff8d13850ceab0 [98519.056995] FS: 0000000000000000(0000) GS:ffff8d15b2480000(0000) knlGS:0000000000000000 [98519.057813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [98519.059480] CR2: 00007fac14a023c8 CR3: 00000001225d9000 CR4: 00000000000006e0 [98519.064479] Call Trace: [98519.064785] ? show_regs.cold.9+0x22/0x2f [98519.069459] ? __die_body+0x22/0x90 [98519.069872] ? die_addr+0x50/0x90 [98519.072390] ? do_general_protection+0x208/0x420 [98519.072938] ? general_protection+0x1e/0x30 [98519.075543] ? __dev_pm_qos_remove_request+0x16/0x210 [98519.076126] ? __might_sleep+0x59/0xc0 [98519.077637] dev_pm_qos_remove_request+0x2d/0x50 [98519.078187] ? process_one_work+0x2c8/0x700 [98519.080795] ? worker_thread+0x296/0x6e0 [98519.081263] ? rescuer_thread+0x570/0x570 [98519.082126] ? kthread+0x1d1/0x200 [98519.082535] ? set_kthread_struct+0x70/0x70 [98519.083018] ? ret_from_fork+0x1f/0x30 [98519.083417] Modules linked in: ldiskfs(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey loop zfs(O) spl(O) ec(O) crc32_generic virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix libata serio_raw dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: ptlrpc]
Looks like the root cause is the LU-18446 that does not properly stop delayed work so it fires into freed memory on module unload at times
There was a previous attempt to fix this in LU-18788 but that turned out to be incomplete either.
