Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
On Master (single node client/master) with 5.14.0-362.13.1.el9_3.x86_64. Plain venilla Rocky9.3. Running just test 77j(only manifests after fresh boot, otherwise it is not seen) gets below refcount_t warning. This is different from what Alex reported on LU-16797 will use this ticket to track Alex's report and this one below
[ 203.338271] ------------[ cut here ]------------ [ 203.338274] refcount_t: addition on 0; use-after-free. [ 203.338289] WARNING: CPU: 1 PID: 4351 at lib/refcount.c:25 refcount_warn_saturate+0x74/0x110 [ 203.341276] Modules linked in: loop dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) sunrpc libcfs(OE) rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm vmwgfx drm_ttm_helper ttm drm_kms_helper pcspkr irqbypass i2c_piix4 syscopyarea sysfillrect sysimgblt drm fuse ext4 mbcache jbd2 sd_mod sr_mod t10_pi cdrom sg ata_generic ahci crct10dif_pclmul crc32_pclmul libahci crc32c_intel ata_piix ghash_clmulni_intel libata e1000 video wmi serio_raw [ 203.349883] CPU: 1 PID: 4351 Comm: ll_ost_io00_002 Kdump: loaded Tainted: G OE ------- --- 5.14.0-362.13.1.el9_3.x86_64 #1 [ 203.352570] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 203.353977] RIP: 0010:refcount_warn_saturate+0x74/0x110 [ 203.354852] Code: 01 01 e8 0f b9 b0 ff 0f 0b e9 28 bf 66 00 80 3d 81 e0 b2 01 00 75 cb 48 c7 c7 70 97 b5 a9 c6 05 71 e0 b2 01 01 e8 ec b8 b0 ff <0f> 0b e9 05 bf 66 00 80 3d 60 e0 b2 01 00 75 a8 48 c7 c7 48 97 b5 [ 203.356648] RSP: 0018:ffff9fda409cbcd0 EFLAGS: 00010282 [ 203.357528] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027 [ 203.358473] RDX: 0000000000000027 RSI: ffffffffaa2671e0 RDI: ffff899abdd1f888 [ 203.359397] RBP: ffff899a71339900 R08: 80000000ffff826f R09: ffff9fda409cbc58 [ 203.360292] R10: 0000000000000001 R11: 000000000000002c R12: ffff899a6b115418 [ 203.361222] R13: ffff899a6d1802b8 R14: 0000000000000002 R15: ffff9fda409cbd30 [ 203.362179] FS: 0000000000000000(0000) GS:ffff899abdd00000(0000) knlGS:0000000000000000 [ 203.363117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 203.363998] CR2: 000055b2d34eb050 CR3: 0000000017bc4000 CR4: 00000000000506e0 [ 203.364896] Call Trace: [ 203.365762] <TASK> [ 203.366631] ? srso_return_thunk+0x5/0x5f [ 203.367507] ? show_trace_log_lvl+0x26e/0x2df [ 203.368352] ? show_trace_log_lvl+0x26e/0x2df [ 203.369178] ? cfs_hash_bd_lookup_intent+0x72/0xd0 [libcfs] [ 203.370033] ? refcount_warn_saturate+0x74/0x110 [ 203.370872] ? __warn+0x81/0x110 [ 203.371669] ? refcount_warn_saturate+0x74/0x110 [ 203.372466] ? report_bug+0x10a/0x140 [ 203.373256] ? handle_bug+0x3c/0x70 [ 203.374067] ? exc_invalid_op+0x14/0x70 [ 203.374843] ? asm_exc_invalid_op+0x16/0x20 [ 203.375571] ? refcount_warn_saturate+0x74/0x110 [ 203.376289] cfs_hash_bd_lookup_intent+0x72/0xd0 [libcfs] [ 203.377012] nrs_tbf_jobid_hash_lookup+0xe/0x70 [ptlrpc] [ 203.377858] nrs_tbf_jobid_cli_find+0x72/0xb0 [ptlrpc] [ 203.378677] nrs_tbf_res_get+0x41/0x550 [ptlrpc] [ 203.379455] nrs_resource_get+0x6e/0xf0 [ptlrpc] [ 203.380216] nrs_resource_get_safe+0xb0/0x170 [ptlrpc] [ 203.380954] ptlrpc_nrs_req_initialize+0x58/0xb0 [ptlrpc] [ 203.381693] ptlrpc_server_request_add+0xfa/0x600 [ptlrpc] [ 203.382452] ptlrpc_server_handle_req_in.isra.0+0x567/0xa20 [ptlrpc] [ 203.383205] ptlrpc_main+0xa35/0xfa0 [ptlrpc] [ 203.383935] ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc] [ 203.384645] kthread+0xe0/0x100 [ 203.385280] ? __pfx_kthread+0x10/0x10 [ 203.385921] ret_from_fork+0x2c/0x50 [ 203.386525] </TASK> [ 203.387115] ---[ end trace 594e1023e886cdf2 ]--- [ 203.387716] ------------[ cut here ]------------ [ 203.387720] refcount_t: saturated; leaking memory. [ 203.387742] WARNING: CPU: 0 PID: 4350 at lib/refcount.c:22 refcount_warn_saturate+0x51/0x110 [ 203.393905] Modules linked in: loop dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) sunrpc libcfs(OE) rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm vmwgfx drm_ttm_helper ttm drm_kms_helper pcspkr irqbypass i2c_piix4 syscopyarea sysfillrect sysimgblt drm fuse ext4 mbcache jbd2 sd_mod sr_mod t10_pi cdrom sg ata_generic ahci crct10dif_pclmul crc32_pclmul libahci crc32c_intel ata_piix ghash_clmulni_intel libata e1000 video wmi serio_raw [ 203.398522] CPU: 0 PID: 4350 Comm: ll_ost_io00_001 Kdump: loaded Tainted: G W OE ------- --- 5.14.0-362.13.1.el9_3.x86_64 #1 [ 203.399279] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 203.399860] RIP: 0010:refcount_warn_saturate+0x51/0x110 [ 203.400418] Code: 84 bc 00 00 00 e9 4f bf 66 00 85 f6 74 46 80 3d a5 e0 b2 01 00 75 ee 48 c7 c7 48 97 b5 a9 c6 05 95 e0 b2 01 01 e8 0f b9 b0 ff <0f> 0b e9 28 bf 66 00 80 3d 81 e0 b2 01 00 75 cb 48 c7 c7 70 97 b5 [ 203.401591] RSP: 0018:ffff9fda409c3cd0 EFLAGS: 00010282 [ 203.402667] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027 [ 203.403259] RDX: 0000000000000027 RSI: ffffffffaa2671e0 RDI: ffff899abdc1f888 [ 203.403865] RBP: ffff899a71339900 R08: 80000000ffff829e R09: ffff9fda409c3c58 [ 203.404448] R10: 0000000000000001 R11: 0000000000000028 R12: ffff899a6b115418 [ 203.405043] R13: ffff899a6d1804a0 R14: 0000000000000002 R15: ffff9fda409c3d30 [ 203.405618] FS: 0000000000000000(0000) GS:ffff899abdc00000(0000) knlGS:0000000000000000 [ 203.406409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 203.407013] CR2: 00007efd7abfd000 CR3: 0000000017bca000 CR4: 00000000000506f0 [ 203.407608] Call Trace: [ 203.408204] <TASK> [ 203.408805] ? srso_return_thunk+0x5/0x5f [ 203.409399] ? show_trace_log_lvl+0x26e/0x2df [ 203.410012] ? show_trace_log_lvl+0x26e/0x2df [ 203.410601] ? cfs_hash_bd_lookup_intent+0x72/0xd0 [libcfs] [ 203.411211] ? refcount_warn_saturate+0x51/0x110 [ 203.411813] ? __warn+0x81/0x110 [ 203.412399] ? refcount_warn_saturate+0x51/0x110 [ 203.413002] ? report_bug+0x10a/0x140 [ 203.413600] ? handle_bug+0x3c/0x70 [ 203.414221] ? exc_invalid_op+0x14/0x70 [ 203.414822] ? asm_exc_invalid_op+0x16/0x20 [ 203.415415] ? refcount_warn_saturate+0x51/0x110 [ 203.416015] cfs_hash_bd_lookup_intent+0x72/0xd0 [libcfs] [ 203.416619] nrs_tbf_jobid_hash_lookup+0xe/0x70 [ptlrpc] [ 203.417479] nrs_tbf_jobid_cli_find+0x72/0xb0 [ptlrpc] [ 203.418257] nrs_tbf_res_get+0x41/0x550 [ptlrpc] [ 203.418960] nrs_resource_get+0x6e/0xf0 [ptlrpc] [ 203.419642] nrs_resource_get_safe+0xb0/0x170 [ptlrpc] [ 203.420329] ptlrpc_nrs_req_initialize+0x58/0xb0 [ptlrpc] [ 203.420997] ptlrpc_server_request_add+0xfa/0x600 [ptlrpc] [ 203.421710] ptlrpc_server_handle_req_in.isra.0+0x567/0xa20 [ptlrpc] [ 203.422390] ptlrpc_main+0xa35/0xfa0 [ptlrpc] [ 203.423057] ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc] [ 203.423751] kthread+0xe0/0x100 [ 203.424362] ? __pfx_kthread+0x10/0x10 [ 203.424953] ret_from_fork+0x2c/0x50 [ 203.425545] </TASK> [ 203.426166] ---[ end trace 594e1023e886cdf3 ]---
Attachments
Issue Links
Activity
Reproducible on 4.18.0-513.9.1.el8_lustre.x86_64
git bisect log | tail -1 # first bad commit: [7375cd6da6329e4d4a4bfb600d4fc9552b3c4c6f] LU-16796 ldlm: Change struct ldlm_resource to use refcount_t
Alex's Reproducer
MDSCOUNT=2 ONLY=77 bash sanityn.sh
With multiple DEBUG_ options enabled in the kernel:
[alexey@rz linux-4.18.0-477.15.1.el8_8]$ grep DEBUG .config|grep -v '^#'
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_BLK_DEBUG_FS=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_LOCK_DEBUGGING_SUPPORT=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_RWSEMS=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_IRQFLAGS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_PI_LIST=y
Alex's Bug Report
[ 3112.751726] ------------[ cut here ]------------ [ 3112.752957] refcount_t hit zero at nrs_tbf_jobid_hop_put+0x10/0x20 [ptlrpc] in mdt00_005[9359], uid/euid: 0/0 [ 3112.753151] WARNING: CPU: 1 PID: 9359 at kernel/panic.c:703 refcount_error_report.cold.9+0x60/0x7b [ 3112.753231] Modules linked in: zfs(O) zunicode(O) zzstd(O) zlua(O) zcommon(O) znvpair(O) zavl(O) icp(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 3112.753574] CPU: 1 PID: 9359 Comm: mdt00_005 Tainted: G W O --------- - - 4.18.0 #11 [ 3112.753650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 3112.753726] RIP: 0010:refcount_error_report.cold.9+0x60/0x7b [ 3112.753782] Code: 25 c0 ce 01 00 48 8b 95 80 00 00 00 48 8d 88 60 0b 00 00 48 c7 c7 88 f4 e1 a8 44 8b 80 20 09 00 00 56 48 89 de e8 21 fc ff ff <0f> 0b 58 e9 16 fc ff ff 8b 35 c1 73 f6 00 eb a8 44 8b 0d b8 73 f6 [ 3112.753940] RSP: 0018:ffffa327e0b83c30 EFLAGS: 00010286 [ 3112.753985] RAX: 0000000000000061 RBX: ffffffffa8e1dbb3 RCX: 0000000000000002 [ 3112.754049] RDX: 0000000000000002 RSI: ffffffffa8e39811 RDI: 00000000ffffffff [ 3112.754114] RBP: ffffa327e0b83cc8 R08: 0000000000000000 R09: 0000000000000000 [ 3112.754179] R10: ffffa327e0b83a78 R11: ffffa327e0b83a70 R12: 0000000000000004 [ 3112.754244] R13: 0000000000000000 R14: ffffa327e0b83cc8 R15: ffffffffc06a365c [ 3112.754309] FS: 0000000000000000(0000) GS:ffffa328b6d00000(0000) knlGS:0000000000000000 [ 3112.754574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3112.754629] CR2: 000056317cd7cf44 CR3: 00000001d5050000 CR4: 0000000000350ea0 [ 3112.754696] Call Trace: [ 3112.755488] ex_handler_refcount+0x49/0x70 [ 3112.756600] fixup_exception+0x2b/0x36 [ 3112.756681] do_trap+0x45/0x110 [ 3112.756719] do_invalid_op+0x23/0x30 [ 3112.756754] ? ptlrpc_exit+0x90/0x114 [ptlrpc] [ 3112.758640] invalid_op+0x14/0x20 [ 3112.758705] RIP: 0010:nrs_tbf_jobid_hop_put+0x10/0x20 [ptlrpc] [ 3112.759088] Code: 66 2e 0f 1f 84 00 00 00 00 00 f0 ff 46 54 0f 88 d8 7c 05 00 c3 0f 1f 44 00 00 f0 ff 4e 54 0f 84 ce 7c 05 00 0f 88 c8 7c 05 00 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 55 48 89 fd 53 48 [ 3112.759238] RSP: 0018:ffffa327e0b83d70 EFLAGS: 00010246 [ 3112.759284] RAX: ffffffffc064bb80 RBX: ffffa328547dba00 RCX: ffffa32852a1e06c [ 3112.759421] RDX: 0000000000000003 RSI: ffffa32852a1e018 RDI: ffffa328547dba00 [ 3112.761099] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 [ 3112.761296] R10: ffffa32855dfd000 R11: ffffa32855dfc1be R12: ffffa32852a1e018 [ 3112.761493] R13: 0000000000000077 R14: ffffffffc00e9a78 R15: 0000000000000000 [ 3112.761560] ? nrs_tbf_id_hop_get+0x10/0x10 [ptlrpc] [ 3112.762286] cfs_hash_putref+0x1ad/0x4d0 [libcfs] [ 3112.762342] nrs_tbf_stop+0x4a/0x2a0 [ptlrpc] [ 3112.762514] nrs_policy_started_put+0x35/0x1d0 [ptlrpc] [ 3112.763111] ptlrpc_nrs_req_finalize+0x1a/0x30 [ptlrpc] [ 3112.764220] ptlrpc_server_finish_active_request+0x48/0x120 [ptlrpc] [ 3112.764359] ptlrpc_main+0x13cf/0x3530 [ptlrpc] [ 3112.764486] ? _raw_spin_unlock_irqrestore+0x43/0x70 [ 3112.764536] ? ptlrpc_wait_event+0x530/0x530 [ptlrpc] [ 3112.765342] kthread+0x16e/0x1a0 [ 3112.765688] ? set_kthread_struct+0x40/0x40 [ 3112.838547] ret_from_fork+0x24/0x30 [ 3112.838594] irq event stamp: 0 [ 3112.838628] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [ 3112.838686] hardirqs last disabled at (0): [<ffffffffa80ddd45>] copy_process+0x4c5/0x1bb0 [ 3112.838756] softirqs last enabled at (0): [<ffffffffa80ddd45>] copy_process+0x4c5/0x1bb0 [ 3112.838893] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 3112.838960] ---[ end trace de302b6ab21f5dde ]--- [ 3112.854924] ------------[ cut here ]------------
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57627
Subject: LU-18606 tests: Test patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f0cf120916e5bb974a8e211804eaa621c4612d38