Details
-
Bug
-
Resolution: Unresolved
-
Medium
-
Lustre 2.17.0
-
None
-
3
-
9223372036854775807
Description
Enabled delayed object release option (DEBUG_KOBJECT_RELEASE) in rocky 8.10 kernel and that makes a crash on any test studown trying to remove the echo module (and may be the others too?)
So need to look into it.
(delayed 2000) [ 1037.771573] kobject: 'echo_client' (0000000061842275): kobject_release, parent 00000000414db884 (delayed 1000) [ 1037.781284] kobject: 'echo_object_kmem' (00000000cb978396): kobject_release, parent 0000000000000000 (delayed 3000) [ 1037.784877] ============================================================================= [ 1037.787425] BUG echo_thread_kmem (Tainted: G O -------- - -): Objects remaining in echo_thread_kmem on __kmem_cache_shutdown() [ 1037.790811] ----------------------------------------------------------------------------- [ 1037.792736] Slab 0x00000000f2143803 objects=26 used=11 fp=0x00000000a59fdd62 flags=0x17ffffc0008100(slab|head|node=0|zone=2|lastcpupid=0x1fffff) [ 1037.797140] CPU: 1 PID: 22352 Comm: rmmod Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #8 [ 1037.800700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 1037.803869] Call Trace: [ 1037.804781] ? dump_stack+0xbb/0x10e [ 1037.806188] ? slab_err+0xa9/0xea [ 1037.807192] ? __might_sleep+0x59/0xc0 [ 1037.808336] ? mutex_lock+0x31/0x70 [ 1037.809325] ? cpumask_next+0x1f/0x30 [ 1037.810410] ? flush_all_cpus_locked+0x187/0x1a0 [ 1037.811748] ? __kmem_cache_shutdown.cold.87+0x48/0x1a7 [ 1037.813218] ? add_timer+0x24/0x50 [ 1037.814216] ? kmem_cache_destroy+0x8d/0x240 [ 1037.815525] ? lu_kmem_fini+0x39/0x80 [obdclass] [ 1037.817055] ? obdecho_exit+0x26/0x5c [obdecho] [ 1037.818407] ? __x64_sys_delete_module+0x1b2/0x3c0 [ 1037.819909] ? do_syscall_64+0xc1/0x440 [ 1037.821020] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1037.822512] Disabling lock debugging due to kernel taint [ 1037.824125] Object 0x000000004139bc7a @offset=4968 [ 1037.825526] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=904259 cpu=2 pid=2356 [ 1037.827725] kmem_cache_alloc+0x3c6/0x430 [ 1037.828555] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.829912] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.830894] lu_env_refill+0x64/0x80 [obdclass] [ 1037.832286] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1037.833609] kthread+0x1d1/0x200 [ 1037.834509] ret_from_fork+0x1f/0x30 [ 1037.835509] Freed in echo_thread_key_fini+0xf8/0x110 [obdecho] age=904286 cpu=0 pid=5491 [ 1037.837831] kmem_cache_free+0x438/0x4d0 [ 1037.839086] echo_thread_key_fini+0xf8/0x110 [obdecho] [ 1037.840758] key_fini+0x5b/0x1e0 [obdclass] [ 1037.842240] lu_context_fini+0x56/0x240 [obdclass] [ 1037.843743] lu_env_fini+0x26/0x40 [obdclass] [ 1037.844921] class_process_config+0x1ede/0x2b60 [obdclass] [ 1037.846487] mgc_apply_recover_logs+0xd20/0x1cf0 [mgc] [ 1037.848301] mgc_process_recover_log+0xb5a/0x1340 [mgc] [ 1037.850076] mgc_process_log+0xb3d/0xbf0 [mgc] [ 1037.851117] mgc_process_config+0x43b/0x1440 [mgc] [ 1037.852643] lustre_process_log+0x5a7/0xda0 [obdclass] [ 1037.854318] ll_fill_super+0x92e/0x15b0 [lustre] [ 1037.855572] lustre_fill_super+0x147/0x5c0 [lustre] [ 1037.857289] mount_nodev+0x56/0xf0 [ 1037.858585] lustre_mount+0x1c/0x30 [lustre] [ 1037.859881] legacy_get_tree+0x2f/0x80 [ 1037.860947] Object 0x0000000025c2328d @offset=7448 [ 1037.862404] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903233 cpu=0 pid=2353 [ 1037.864230] kmem_cache_alloc+0x3c6/0x430 [ 1037.864826] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.866592] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.868287] lu_env_refill+0x64/0x80 [obdclass] [ 1037.869619] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1037.871129] kthread+0x1d1/0x200 [ 1037.872200] ret_from_fork+0x1f/0x30 [ 1037.873325] Object 0x00000000600efa8a @offset=11168 [ 1037.874435] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903209 cpu=1 pid=5491 [ 1037.876526] kmem_cache_alloc+0x3c6/0x430 [ 1037.877634] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.879129] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.880634] lu_env_refill+0x64/0x80 [obdclass] [ 1037.882068] cl_site_init+0xdc/0x130 [obdclass] [ 1037.883573] vvp_device_alloc+0x1de/0x570 [lustre] [ 1037.885313] cl_type_setup+0x53/0x670 [obdclass] [ 1037.887106] cl_sb_init+0xe5/0x130 [lustre] [ 1037.888643] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1037.890468] ll_fill_super+0x1157/0x15b0 [lustre] [ 1037.892065] lustre_fill_super+0x147/0x5c0 [lustre] [ 1037.893617] mount_nodev+0x56/0xf0 [ 1037.894632] lustre_mount+0x1c/0x30 [lustre] [ 1037.896016] legacy_get_tree+0x2f/0x80 [ 1037.897179] vfs_get_tree+0x27/0x130 [ 1037.898304] do_mount+0xc17/0x1030 [ 1037.899323] Object 0x000000008639adf9 @offset=13648 [ 1037.900977] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903233 cpu=0 pid=2354 [ 1037.903847] kmem_cache_alloc+0x3c6/0x430 [ 1037.905072] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.906551] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.907821] lu_env_refill+0x64/0x80 [obdclass] [ 1037.909000] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1037.910725] kthread+0x1d1/0x200 [ 1037.911836] ret_from_fork+0x1f/0x30 [ 1037.913116] Object 0x00000000d75f0096 @offset=14888 [ 1037.914751] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903209 cpu=1 pid=5491 [ 1037.917615] kmem_cache_alloc+0x3c6/0x430 [ 1037.918801] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.920326] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.921630] lu_env_refill+0x64/0x80 [obdclass] [ 1037.922896] cl_site_init+0xdc/0x130 [obdclass] [ 1037.924123] vvp_device_alloc+0x1de/0x570 [lustre] [ 1037.925685] cl_type_setup+0x53/0x670 [obdclass] [ 1037.927244] cl_sb_init+0xe5/0x130 [lustre] [ 1037.928371] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1037.929875] ll_fill_super+0x1157/0x15b0 [lustre] [ 1037.931423] lustre_fill_super+0x147/0x5c0 [lustre] [ 1037.933041] mount_nodev+0x56/0xf0 [ 1037.934133] lustre_mount+0x1c/0x30 [lustre] [ 1037.935589] legacy_get_tree+0x2f/0x80 [ 1037.937064] vfs_get_tree+0x27/0x130 [ 1037.938343] do_mount+0xc17/0x1030 [ 1037.939599] Object 0x00000000cf9d633b @offset=17368 [ 1037.941127] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903209 cpu=1 pid=5491 [ 1037.943542] kmem_cache_alloc+0x3c6/0x430 [ 1037.944759] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.946581] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.948247] lu_env_refill+0x64/0x80 [obdclass] [ 1037.949869] cl_site_init+0xdc/0x130 [obdclass] [ 1037.951318] vvp_device_alloc+0x1de/0x570 [lustre] [ 1037.952865] cl_type_setup+0x53/0x670 [obdclass] [ 1037.954445] cl_sb_init+0xe5/0x130 [lustre] [ 1037.955948] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1037.957823] ll_fill_super+0x1157/0x15b0 [lustre] [ 1037.959309] lustre_fill_super+0x147/0x5c0 [lustre] [ 1037.960465] mount_nodev+0x56/0xf0 [ 1037.961115] lustre_mount+0x1c/0x30 [lustre] [ 1037.962259] legacy_get_tree+0x2f/0x80 [ 1037.962948] vfs_get_tree+0x27/0x130 [ 1037.963845] do_mount+0xc17/0x1030 [ 1037.964612] Object 0x00000000bced4013 @offset=18608 [ 1037.965704] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=904259 cpu=3 pid=2355 [ 1037.967580] kmem_cache_alloc+0x3c6/0x430 [ 1037.968458] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.969616] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.970826] lu_env_refill+0x64/0x80 [obdclass] [ 1037.971977] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1037.973111] kthread+0x1d1/0x200 [ 1037.974021] ret_from_fork+0x1f/0x30 [ 1037.974688] Object 0x000000003b79f6b3 @offset=21088 [ 1037.975604] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=903209 cpu=1 pid=5491 [ 1037.977572] kmem_cache_alloc+0x3c6/0x430 [ 1037.978661] echo_thread_key_init+0x31/0x160 [obdecho] [ 1037.980169] keys_fill+0xb4/0x1b0 [obdclass] [ 1037.981904] lu_env_refill+0x64/0x80 [obdclass] [ 1037.983601] cl_site_init+0xdc/0x130 [obdclass] [ 1037.985312] vvp_device_alloc+0x1de/0x570 [lustre] [ 1037.987070] cl_type_setup+0x53/0x670 [obdclass] [ 1037.988617] cl_sb_init+0xe5/0x130 [lustre] [ 1037.990053] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1037.991770] ll_fill_super+0x1157/0x15b0 [lustre] [ 1037.993487] lustre_fill_super+0x147/0x5c0 [lustre] [ 1037.995265] mount_nodev+0x56/0xf0 [ 1037.996241] lustre_mount+0x1c/0x30 [lustre] [ 1037.997494] legacy_get_tree+0x2f/0x80 [ 1037.998545] vfs_get_tree+0x27/0x130 [ 1037.999480] do_mount+0xc17/0x1030 [ 1038.001124] Object 0x00000000b71a5bba @offset=22328 [ 1038.002737] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=904343 cpu=2 pid=5509 [ 1038.005472] kmem_cache_alloc+0x3c6/0x430 [ 1038.006764] echo_thread_key_init+0x31/0x160 [obdecho] [ 1038.008527] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.010214] lu_site_init_finish+0xc0/0xe0 [obdclass] [ 1038.012005] lov_device_alloc+0x73c/0xa40 [lov] [ 1038.013672] obd_setup+0x2e6/0x700 [obdclass] [ 1038.015346] class_setup+0x6f5/0x9f0 [obdclass] [ 1038.016955] class_process_config+0x1658/0x2b60 [obdclass] [ 1038.018817] class_config_llog_handler+0xc51/0x1af0 [obdclass] [ 1038.020932] llog_process_thread+0x13be/0x2220 [obdclass] [ 1038.022846] llog_process_thread_daemonize+0xa2/0xe0 [obdclass] [ 1038.024905] kthread+0x1d1/0x200 [ 1038.025937] ret_from_fork+0x1f/0x30 [ 1038.027082] Object 0x000000002fcdf895 @offset=23568 [ 1038.028651] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=972595 cpu=3 pid=2903 [ 1038.031422] kmem_cache_alloc+0x3c6/0x430 [ 1038.032638] echo_thread_key_init+0x31/0x160 [obdecho] [ 1038.034330] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.035816] lu_context_init+0xc8/0x240 [obdclass] [ 1038.037419] lu_env_init+0x1e/0x40 [obdclass] [ 1038.038944] cl_env_new+0x141/0x310 [obdclass] [ 1038.040248] cl_env_alloc+0x29/0xe0 [obdclass] [ 1038.041136] 0xffffffffc1bab423 [ 1038.042187] do_one_initcall+0x73/0x360 [ 1038.043469] do_init_module+0xa4/0x350 [ 1038.044580] load_module+0x1c42/0x2260 [ 1038.045330] __do_sys_finit_module+0x141/0x1a0 [ 1038.046115] __x64_sys_finit_module+0x1e/0x30 [ 1038.047171] do_syscall_64+0xc1/0x440 [ 1038.047838] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1038.049279] Object 0x0000000082499be6 @offset=26048 [ 1038.050366] Allocated in echo_thread_key_init+0x31/0x160 [obdecho] age=904599 cpu=0 pid=2352 [ 1038.052614] kmem_cache_alloc+0x3c6/0x430 [ 1038.053563] echo_thread_key_init+0x31/0x160 [obdecho] [ 1038.054718] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.056074] lu_env_refill+0x64/0x80 [obdclass] [ 1038.057694] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.059420] kthread+0x1d1/0x200 [ 1038.060473] ret_from_fork+0x1f/0x30 [ 1038.061909] ------------[ cut here ]------------ [ 1038.063260] kmem_cache_destroy echo_thread_kmem: Slab cache still has objects when called from lu_kmem_fini+0x39/0x80 [obdclass] [ 1038.063463] WARNING: CPU: 1 PID: 22352 at mm/slab_common.c:523 kmem_cache_destroy+0x220/0x240 [ 1038.069651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mgs(O) lquota(O) lfsck(O) obdecho(O-) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common nfsv4 sb_edac rapl dns_resolver i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: mdd] [ 1038.093713] CPU: 1 PID: 22352 Comm: rmmod Kdump: loaded Tainted: G B O -------- - - 4.18.0rh8.10-debug #8 [ 1038.097086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 1038.099450] RIP: 0010:kmem_cache_destroy+0x220/0x240 [ 1038.101398] Code: bd 85 dc 02 01 48 83 05 55 86 dc 02 01 48 83 05 55 86 dc 02 01 48 83 05 55 86 dc 02 01 e8 14 16 d4 ff 48 83 05 50 86 dc 02 01 <0f> 0b 48 83 05 4e 86 dc 02 01 48 83 05 4e 86 dc 02 01 e9 14 fe ff [ 1038.107610] RSP: 0018:ffffb58dc1fefe98 EFLAGS: 00010202 [ 1038.108815] RAX: 0000000000000000 RBX: ffff93c1112d8440 RCX: 0000000000000001 [ 1038.111448] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff93c14209e660 [ 1038.114330] RBP: 0000000080010d00 R08: 0000000000000000 R09: c0000000ffff7fff [ 1038.116942] R10: 0000000000000001 R11: ffffb58dc1fefc88 R12: 0000000000000000 [ 1038.119795] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1038.122216] FS: 00007f508ddb3740(0000) GS:ffff93c142080000(0000) knlGS:0000000000000000 [ 1038.124865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1038.126842] CR2: 000056159fba09f8 CR3: 000000011832c006 CR4: 0000000000170ee0 [ 1038.129016] Call Trace: [ 1038.129564] ? show_regs.cold.9+0x22/0x2f [ 1038.131067] ? __warn+0xc8/0x150 [ 1038.132304] ? kmem_cache_destroy+0x220/0x240 [ 1038.133525] ? report_bug+0x113/0x140 [ 1038.135061] ? do_error_trap+0xb6/0x130 [ 1038.136702] ? do_invalid_op+0x46/0x60 [ 1038.138259] ? kmem_cache_destroy+0x220/0x240 [ 1038.140017] ? invalid_op+0x14/0x20 [ 1038.141305] ? kmem_cache_destroy+0x220/0x240 [ 1038.143183] ? kmem_cache_destroy+0x218/0x240 [ 1038.144976] lu_kmem_fini+0x39/0x80 [obdclass] [ 1038.146866] obdecho_exit+0x26/0x5c [obdecho] [ 1038.148426] __x64_sys_delete_module+0x1b2/0x3c0 [ 1038.149924] do_syscall_64+0xc1/0x440 [ 1038.150920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1038.152534] RIP: 0033:0x7f508ccb2bab [ 1038.153784] Code: 73 01 c3 48 8b 0d dd 62 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 62 39 00 f7 d8 64 89 01 48 [ 1038.158801] RSP: 002b:00007fff5b189638 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 1038.160846] RAX: ffffffffffffffda RBX: 000056159fb967c0 RCX: 00007f508ccb2bab [ 1038.162662] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000056159fb96828 [ 1038.165341] RBP: 0000000000000000 R08: 00007fff5b1885b1 R09: 0000000000000000 [ 1038.167825] R10: 00007f508cdf9e80 R11: 0000000000000206 R12: 00007fff5b189850 [ 1038.170698] R13: 00007fff5b18af6d R14: 000056159fb962a0 R15: 000056159fb967c0 [ 1038.172850] ---[ end trace 2fa53f6a4899e151 ]--- [ 1038.174258] ============================================================================= [ 1038.176325] BUG echo_session_kmem (Tainted: G B W O -------- - -): Objects remaining in echo_session_kmem on __kmem_cache_shutdown() [ 1038.179777] ----------------------------------------------------------------------------- [ 1038.182524] Slab 0x000000003d7ab741 objects=23 used=10 fp=0x00000000baeb1cde flags=0x17ffffc0008100(slab|head|node=0|zone=2|lastcpupid=0x1fffff) [ 1038.186631] CPU: 1 PID: 22352 Comm: rmmod Kdump: loaded Tainted: G B W O -------- - - 4.18.0rh8.10-debug #8 [ 1038.190841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 1038.194331] Call Trace: [ 1038.195344] ? dump_stack+0xbb/0x10e [ 1038.196746] ? slab_err+0xa9/0xea [ 1038.197822] ? __might_sleep+0x59/0xc0 [ 1038.199044] ? mutex_lock+0x31/0x70 [ 1038.200270] ? cpumask_next+0x1f/0x30 [ 1038.201682] ? flush_all_cpus_locked+0x187/0x1a0 [ 1038.203216] ? __kmem_cache_shutdown.cold.87+0x48/0x1a7 [ 1038.205292] ? kmem_cache_destroy+0x8d/0x240 [ 1038.207055] ? lu_kmem_fini+0x39/0x80 [obdclass] [ 1038.209182] ? obdecho_exit+0x26/0x5c [obdecho] [ 1038.210927] ? __x64_sys_delete_module+0x1b2/0x3c0 [ 1038.212047] ? do_syscall_64+0xc1/0x440 [ 1038.213027] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1038.214669] Object 0x00000000ba7e266a @offset=8 [ 1038.215854] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903598 cpu=1 pid=5491 [ 1038.218300] kmem_cache_alloc+0x3c6/0x430 [ 1038.219731] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.221840] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.223799] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.225725] cl_site_init+0xdc/0x130 [obdclass] [ 1038.227727] vvp_device_alloc+0x1de/0x570 [lustre] [ 1038.229678] cl_type_setup+0x53/0x670 [obdclass] [ 1038.231132] cl_sb_init+0xe5/0x130 [lustre] [ 1038.232567] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1038.234659] ll_fill_super+0x1157/0x15b0 [lustre] [ 1038.236477] lustre_fill_super+0x147/0x5c0 [lustre] [ 1038.238345] mount_nodev+0x56/0xf0 [ 1038.239596] lustre_mount+0x1c/0x30 [lustre] [ 1038.240932] legacy_get_tree+0x2f/0x80 [ 1038.242322] vfs_get_tree+0x27/0x130 [ 1038.243585] do_mount+0xc17/0x1030 [ 1038.244746] Object 0x00000000de60cffc @offset=352 [ 1038.246319] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903623 cpu=0 pid=2353 [ 1038.249403] kmem_cache_alloc+0x3c6/0x430 [ 1038.251018] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.252773] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.254220] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.255497] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.257470] kthread+0x1d1/0x200 [ 1038.258772] ret_from_fork+0x1f/0x30 [ 1038.260255] Object 0x00000000b6245327 @offset=2072 [ 1038.261575] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903623 cpu=0 pid=2354 [ 1038.264653] kmem_cache_alloc+0x3c6/0x430 [ 1038.266012] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.267962] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.269177] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.270303] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.271591] kthread+0x1d1/0x200 [ 1038.272629] ret_from_fork+0x1f/0x30 [ 1038.273877] Object 0x00000000013f857a @offset=2416 [ 1038.275069] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903599 cpu=1 pid=5491 [ 1038.277473] kmem_cache_alloc+0x3c6/0x430 [ 1038.279009] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.280383] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.282336] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.284062] cl_site_init+0xdc/0x130 [obdclass] [ 1038.285983] vvp_device_alloc+0x1de/0x570 [lustre] [ 1038.287154] cl_type_setup+0x53/0x670 [obdclass] [ 1038.289044] cl_sb_init+0xe5/0x130 [lustre] [ 1038.290585] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1038.292300] ll_fill_super+0x1157/0x15b0 [lustre] [ 1038.294129] lustre_fill_super+0x147/0x5c0 [lustre] [ 1038.296247] mount_nodev+0x56/0xf0 [ 1038.297673] lustre_mount+0x1c/0x30 [lustre] [ 1038.299485] legacy_get_tree+0x2f/0x80 [ 1038.301042] vfs_get_tree+0x27/0x130 [ 1038.302376] do_mount+0xc17/0x1030 [ 1038.303653] Object 0x00000000a939152c @offset=3104 [ 1038.305456] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903599 cpu=1 pid=5491 [ 1038.308641] kmem_cache_alloc+0x3c6/0x430 [ 1038.310147] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.312057] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.313899] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.315700] cl_site_init+0xdc/0x130 [obdclass] [ 1038.316846] vvp_device_alloc+0x1de/0x570 [lustre] [ 1038.318505] cl_type_setup+0x53/0x670 [obdclass] [ 1038.320444] cl_sb_init+0xe5/0x130 [lustre] [ 1038.322173] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1038.324339] ll_fill_super+0x1157/0x15b0 [lustre] [ 1038.326195] lustre_fill_super+0x147/0x5c0 [lustre] [ 1038.328137] mount_nodev+0x56/0xf0 [ 1038.329501] lustre_mount+0x1c/0x30 [lustre] [ 1038.331175] legacy_get_tree+0x2f/0x80 [ 1038.332578] vfs_get_tree+0x27/0x130 [ 1038.333847] do_mount+0xc17/0x1030 [ 1038.335080] Object 0x00000000f8c4172d @offset=3448 [ 1038.336720] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=903599 cpu=1 pid=5491 [ 1038.339082] kmem_cache_alloc+0x3c6/0x430 [ 1038.340551] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.342464] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.344217] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.345371] cl_site_init+0xdc/0x130 [obdclass] [ 1038.346289] vvp_device_alloc+0x1de/0x570 [lustre] [ 1038.347450] cl_type_setup+0x53/0x670 [obdclass] [ 1038.348850] cl_sb_init+0xe5/0x130 [lustre] [ 1038.349840] client_common_fill_super+0x3ec6/0x5210 [lustre] [ 1038.351401] ll_fill_super+0x1157/0x15b0 [lustre] [ 1038.352718] lustre_fill_super+0x147/0x5c0 [lustre] [ 1038.354032] mount_nodev+0x56/0xf0 [ 1038.355324] lustre_mount+0x1c/0x30 [lustre] [ 1038.357021] legacy_get_tree+0x2f/0x80 [ 1038.358491] vfs_get_tree+0x27/0x130 [ 1038.359524] do_mount+0xc17/0x1030 [ 1038.360616] Object 0x00000000b708b0d7 @offset=4136 [ 1038.362002] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=972754 cpu=3 pid=2903 [ 1038.364446] kmem_cache_alloc+0x3c6/0x430 [ 1038.365575] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.367565] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.369138] lu_context_init+0xc8/0x240 [obdclass] [ 1038.371080] cl_env_new+0x254/0x310 [obdclass] [ 1038.373076] cl_env_alloc+0x29/0xe0 [obdclass] [ 1038.375360] 0xffffffffc1bab423 [ 1038.377473] do_one_initcall+0x73/0x360 [ 1038.379461] do_init_module+0xa4/0x350 [ 1038.380853] load_module+0x1c42/0x2260 [ 1038.382180] __do_sys_finit_module+0x141/0x1a0 [ 1038.383374] __x64_sys_finit_module+0x1e/0x30 [ 1038.384711] do_syscall_64+0xc1/0x440 [ 1038.385841] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1038.387300] Object 0x00000000627c92c5 @offset=5168 [ 1038.388783] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=904758 cpu=0 pid=2352 [ 1038.390990] kmem_cache_alloc+0x3c6/0x430 [ 1038.391962] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.393243] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.394869] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.396375] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.398015] kthread+0x1d1/0x200 [ 1038.399110] ret_from_fork+0x1f/0x30 [ 1038.400217] Freed in echo_session_key_fini+0xd7/0xf0 [obdecho] age=904759 cpu=0 pid=5491 [ 1038.402689] kmem_cache_free+0x438/0x4d0 [ 1038.403566] echo_session_key_fini+0xd7/0xf0 [obdecho] [ 1038.405171] key_fini+0x5b/0x1e0 [obdclass] [ 1038.406354] lu_context_fini+0x56/0x240 [obdclass] [ 1038.408281] obd_setup+0x101/0x700 [obdclass] [ 1038.410004] class_setup+0x6f5/0x9f0 [obdclass] [ 1038.411373] class_process_config+0x1658/0x2b60 [obdclass] [ 1038.413280] do_lcfg+0x376/0x740 [obdclass] [ 1038.415066] lustre_start_simple+0x8f/0x220 [obdclass] [ 1038.417005] lustre_start_mgc+0x145a/0x29b0 [obdclass] [ 1038.418802] lustre_fill_super+0x12d/0x5c0 [lustre] [ 1038.420521] mount_nodev+0x56/0xf0 [ 1038.421587] lustre_mount+0x1c/0x30 [lustre] [ 1038.423063] legacy_get_tree+0x2f/0x80 [ 1038.424174] vfs_get_tree+0x27/0x130 [ 1038.425277] do_mount+0xc17/0x1030 [ 1038.426368] Object 0x000000000de2dd16 @offset=6200 [ 1038.427963] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=904648 cpu=3 pid=2355 [ 1038.430583] kmem_cache_alloc+0x3c6/0x430 [ 1038.431725] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.433283] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.434867] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.436466] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.438124] kthread+0x1d1/0x200 [ 1038.439184] ret_from_fork+0x1f/0x30 [ 1038.440359] Object 0x00000000724b6e75 @offset=6544 [ 1038.441928] Allocated in echo_session_key_init+0x31/0x160 [obdecho] age=904649 cpu=2 pid=2356 [ 1038.444658] kmem_cache_alloc+0x3c6/0x430 [ 1038.445912] echo_session_key_init+0x31/0x160 [obdecho] [ 1038.447548] keys_fill+0xb4/0x1b0 [obdclass] [ 1038.448714] lu_env_refill+0x4d/0x80 [obdclass] [ 1038.449591] ptlrpcd+0x325/0xe40 [ptlrpc] [ 1038.450522] kthread+0x1d1/0x200 [ 1038.451212] ret_from_fork+0x1f/0x30 [ 1038.451917] Freed in echo_session_key_fini+0xd7/0xf0 [obdecho] age=904700 cpu=1 pid=5509 [ 1038.454153] kmem_cache_free+0x438/0x4d0 [ 1038.455191] echo_session_key_fini+0xd7/0xf0 [obdecho] [ 1038.456275] key_fini+0x5b/0x1e0 [obdclass] [ 1038.457671] lu_context_fini+0x56/0x240 [obdclass] [ 1038.458895] obd_setup+0x101/0x700 [obdclass] [ 1038.460044] class_setup+0x6f5/0x9f0 [obdclass] [ 1038.461340] class_process_config+0x1658/0x2b60 [obdclass] [ 1038.462757] class_config_llog_handler+0xc51/0x1af0 [obdclass] [ 1038.464505] llog_process_thread+0x13be/0x2220 [obdclass] [ 1038.466527] llog_process_thread_daemonize+0xa2/0xe0 [obdclass] [ 1038.467779] kthread+0x1d1/0x200 [ 1038.468496] ret_from_fork+0x1f/0x30 [ 1038.469911] ------------[ cut here ]------------ [ 1038.471449] kmem_cache_destroy echo_session_kmem: Slab cache still has objects when called from lu_kmem_fini+0x39/0x80 [obdclass] [ 1038.471644] WARNING: CPU: 1 PID: 22352 at mm/slab_common.c:523 kmem_cache_destroy+0x220/0x240 [ 1038.476864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mgs(O) lquota(O) lfsck(O) obdecho(O-) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common nfsv4 sb_edac rapl dns_resolver i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: mdd] [ 1038.490476] CPU: 1 PID: 22352 Comm: rmmod Kdump: loaded Tainted: G B W O -------- - - 4.18.0rh8.10-debug #8 [ 1038.493633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 1038.496190] RIP: 0010:kmem_cache_destroy+0x220/0x240 [ 1038.497684] Code: bd 85 dc 02 01 48 83 05 55 86 dc 02 01 48 83 05 55 86 dc 02 01 48 83 05 55 86 dc 02 01 e8 14 16 d4 ff 48 83 05 50 86 dc 02 01 <0f> 0b 48 83 05 4e 86 dc 02 01 48 83 05 4e 86 dc 02 01 e9 14 fe ff [ 1038.502791] RSP: 0018:ffffb58dc1fefe98 EFLAGS: 00010202 [ 1038.504394] RAX: 0000000000000000 RBX: ffff93c1112d8c40 RCX: 0000000000000001 [ 1038.506458] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff93c14209e660 [ 1038.508524] RBP: 0000000080010d00 R08: 0000000000000000 R09: c0000000ffff7fff [ 1038.510592] R10: 0000000000000001 R11: ffffb58dc1fefc88 R12: 0000000000000000 [ 1038.512831] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1038.515036] FS: 00007f508ddb3740(0000) GS:ffff93c142080000(0000) knlGS:0000000000000000 [ 1038.517497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1038.519264] CR2: 000056159fba09f8 CR3: 000000011832c006 CR4: 0000000000170ee0 [ 1038.521421] Call Trace: [ 1038.522191] ? show_regs.cold.9+0x22/0x2f [ 1038.523372] ? __warn+0xc8/0x150 [ 1038.524375] ? kmem_cache_destroy+0x220/0x240 [ 1038.525694] ? report_bug+0x113/0x140 [ 1038.526829] ? do_error_trap+0xb6/0x130 [ 1038.528014] ? do_invalid_op+0x46/0x60 [ 1038.529110] ? kmem_cache_destroy+0x220/0x240 [ 1038.530415] ? invalid_op+0x14/0x20 [ 1038.531491] ? kmem_cache_destroy+0x220/0x240 [ 1038.532757] ? kmem_cache_destroy+0x218/0x240 [ 1038.534127] lu_kmem_fini+0x39/0x80 [obdclass] [ 1038.535784] obdecho_exit+0x26/0x5c [obdecho] [ 1038.537121] __x64_sys_delete_module+0x1b2/0x3c0 [ 1038.538559] do_syscall_64+0xc1/0x440 [ 1038.539665] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1038.541169] RIP: 0033:0x7f508ccb2bab [ 1038.542262] Code: 73 01 c3 48 8b 0d dd 62 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ad 62 39 00 f7 d8 64 89 01 48 [ 1038.547850] RSP: 002b:00007fff5b189638 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 1038.550176] RAX: ffffffffffffffda RBX: 000056159fb967c0 RCX: 00007f508ccb2bab [ 1038.552473] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000056159fb96828 [ 1038.554309] RBP: 0000000000000000 R08: 00007fff5b1885b1 R09: 0000000000000000 [ 1038.555435] R10: 00007f508cdf9e80 R11: 0000000000000206 R12: 00007fff5b189850 [ 1038.557145] R13: 00007fff5b18af6d R14: 000056159fb962a0 R15: 000056159fb967c0 [ 1038.558404] ---[ end trace 2fa53f6a4899e152 ]--- [ 1038.559762] kobject: 'obdecho' (00000000a6f97714): kobject_release, parent 00000000414db884 (delayed 1000) [ 1038.565944] kobject: 'holders' (00000000afe9b820): kobject_release, parent 00000000be453869 (delayed 4000) [ 1038.568126] kobject: 'notes' (000000000b4d3bcf): kobject_release, parent 00000000be453869 (delayed 2000) [ 1038.815297] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 [ 1038.817995] PGD 0 P4D 0 [ 1038.818749] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 1038.820100] CPU: 1 PID: 21450 Comm: kworker/1:2 Kdump: loaded Tainted: G B W O -------- - - 4.18.0rh8.10-debug #8 [ 1038.823307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 1038.825934] Workqueue: events kobject_delayed_cleanup [ 1038.827412] RIP: 0010:kmem_cache_free+0x2d6/0x4d0 [ 1038.828702] Code: 03 48 83 05 0b db db 02 01 48 85 c0 75 dd 48 83 05 06 db db 02 01 48 83 05 06 db db 02 01 65 ff 0d a7 73 1a 4a e9 2a ff ff ff <48> 8b 57 58 49 8b 4e 58 48 c7 c6 40 79 e4 b6 48 c7 c7 70 e1 3a b7 [ 1038.834110] RSP: 0018:ffffb58dc0d7fcd8 EFLAGS: 00010207 [ 1038.835667] RAX: ffffdfcd44208a00 RBX: ffff93c18822dc10 RCX: 0017ffffc0000100 [ 1038.837332] RDX: ffffdfcd40000000 RSI: ffff93c120021a60 RDI: 0000000000000000 [ 1038.838638] RBP: ffff93c10822dc10 R08: 0000000000000000 R09: 0000000000000000 [ 1038.839918] R10: ffffffffb82a06ac R11: 0000000000000021 R12: 0000000080000000 [ 1038.841241] R13: ffff93c10916aa20 R14: ffff93c1112d8440 R15: 0000000000000000 [ 1038.842459] FS: 0000000000000000(0000) GS:ffff93c142080000(0000) knlGS:0000000000000000 [ 1038.843661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1038.844593] CR2: 0000000000000058 CR3: 000000000c816002 CR4: 0000000000170ee0 [ 1038.845743] Call Trace: [ 1038.846342] ? show_regs.cold.9+0x22/0x2f [ 1038.847050] ? __die_body+0x22/0x90 [ 1038.847677] ? __die+0x33/0x4a [ 1038.848218] ? no_context+0x30f/0x5a0 [ 1038.848983] ? __bad_area_nosemaphore+0x1c6/0x260 [ 1038.849884] ? bad_area_nosemaphore+0x1a/0x30 [ 1038.850652] ? do_user_addr_fault+0x540/0x8a0 [ 1038.851359] ? unwind_next_frame+0x102/0x7f0 [ 1038.852180] ? __do_page_fault+0x6b/0xa0 [ 1038.853013] ? do_page_fault+0x87/0x30f [ 1038.853687] ? page_fault+0x1e/0x30 [ 1038.854282] ? kmem_cache_free+0x2d6/0x4d0 [ 1038.855086] echo_thread_key_fini+0xf8/0x110 [obdecho] [ 1038.856119] key_fini+0x5b/0x1e0 [obdclass] [ 1038.856995] lu_context_key_quiesce+0xb6/0x140 [obdclass] [ 1038.858045] lu_context_key_degister+0x3d/0x1a0 [obdclass] [ 1038.859171] ? __dentry_kill+0x1e2/0x2f0 [ 1038.860033] ? cpumask_next+0x1f/0x30 [ 1038.860873] ? do_raw_spin_unlock+0x75/0x190 [ 1038.861612] lu_context_key_degister_many+0x7c/0xd0 [obdclass] [ 1038.862758] echo_type_fini+0x22/0x30 [obdecho] [ 1038.863668] lu_device_type_fini+0x1c/0x30 [obdclass] [ 1038.865444] class_sysfs_release+0x41/0x1a0 [obdclass] [ 1038.867169] kobject_delayed_cleanup+0xc4/0x310 [ 1038.868515] process_one_work+0x2c8/0x700 [ 1038.869785] worker_thread+0x296/0x6e0 [ 1038.870960] ? rescuer_thread+0x570/0x570 [ 1038.872162] kthread+0x1d1/0x200 [ 1038.873174] ? set_kthread_struct+0x70/0x70 [ 1038.874421] ret_from_fork+0x1f/0x30 [ 1038.876501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mgs(O) lquota(O) lfsck(O) obdecho(O-) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common nfsv4 sb_edac rapl dns_resolver i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: obdecho] [ 1038.902579] CR2: 0000000000000058
the run: http://testing.linuxhacker.ru/lustre-reports/rocky8/133/results.html