[LU-16796] Replace LASSERT_ATOMIC_* (atomic_t) with reference counters (refcount)

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: lug23dd
• Severity: 3
• Assignee: Arshad Hussain
• Reporter: Arshad Hussain

Description

 

For Example.

struct ptlrpc_sec {      
....
atomic_t ps_refcount;    
};

Which is called as...

LASSERT_ATOMIC_ZERO(&sec->ps_refcount);

Now becomes

struct ptlrpc_sec {      
....
struct kref ps_refcount;
};

and every thing should follow to support the above.

 

Currently these are under active use:

LASSERT_ATOMIC_GT
LASSERT_ATOMIC_GT_LT
LASSERT_ATOMIC_ZERO
LASSERT_ATOMIC_POS

Issue Links

• is related to: LU-18606 refcount_t warning when running sanityn/77j (refcount_t: addition on 0; use-after-free.)
• is related to: LU-9859 libcfs simplification

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56195/
Subject: LU-16796 lfsck: Change lfsck_instance to use refcount_t
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2a45af9061cf08acfd965d4ad043b61515b9680a

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53459/
Subject: LU-16796 ptlrpc: Change struct ptlrpc_request_set to use kref
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b00779bf2dbf783bc7a7dcb9ebd5601ab5cddaca

[Arshad Hussain]

Status of work left for this ticket.

Initialization value is not 1 (No change required)

ptlrpc/sec.c:                rq_refcount
ptlrpc/ptlrpc_internal.h:    rq_refcount
ptlrpc/sec_plain.c:          cc_refcount
mgs/mgs_llog.c:              fsdb_ref
ldlm/ldlm_pool.c:            pl_lock_volume_factor
llite/pcc.c:                 pcci_refcount

Along with kref'ed structure extra data-structures reference is called in release function (Will be done last, requires major code change or cannot be done)

osp/osp_trans.c:            ot_refcount
obdclass/obd_mount.c:       cl_mgc_refcount
obdclass/lu_object.c:       loh_ref
lfsck/lfsck_striped_dir.c:  ll_ref
lfsck/lfsck_striped_dir.c:  ll_ref
lfsck/lfsck_layout.c:       lc_ref
lfsck/lfsck_namespace.c:    lc_ref
lfsck/lfsck_engine.c:       ll_ref
 

Does not do any memory release. No change required (No change necessary)

ptlrpc/connection.c:        c_refcount
quota/qsd_lib.c:            qqi_ref
obdclass/cl_io.c:           csi_complete
obdclass/lu_object.c:       lct_used

Todo:(investigation left)

llite/statahead.c:                sax_refcount
llite/statahead.c:                sai_refcount
ptlrpc/gss/sec_gss.c:             src_base.sc_refcount
obdclass/lprocfs_status_server.c: nid_exp_ref_count
obdclass/llog_obd.c:              loc_refcount

WIP

include/obd.h:              mod_refcount

 

 

 

 

 

[Gerrit Updater]

"Arshad Hussain <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58309
Subject: LU-16796 obdclass: Change struct local_oid_storage to use kref
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8fa0bdae73fe73524107dd6d7fad638d83c02fb6

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56454/
Subject: LU-16796 quota: Change struct lquota_entry to use kref
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 01d10c84ad64bfbfe60c45267e9ac9b441191025

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56193/
Subject: LU-16796 ptlrpc: Change struct nrs_tbf_rule to use kref
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 308c7a553809695d475bbb640ffe1c86efd9719f

[Arshad Hussain]

This can be seen on Master with 5.14.0-362.13.1.el9_3.x86_64. Plain venilla Rocky9.3. Running just test 77j(only gets after fresh boot, otherwise it is not seen) gets below refcount_t warning. This is different from what you are seeing on 4.18 kernel at least i can reproduce this warning. Opened LU-18606 for both refcount_t error/warning.

[Arshad Hussain]

Thanks! Looking into this.

[Alex Zhuravlev]

trivial:

MDSCOUNT=2  ONLY=77 bash sanityn.sh

but I do have multiple DEBUG_ options enabled in the kernel:

[alexey@rz linux-4.18.0-477.15.1.el8_8]$ grep DEBUG .config|grep -v '^#'
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_BLK_DEBUG_FS=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_LOCK_DEBUGGING_SUPPORT=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y
CONFIG_DEBUG_RWSEMS=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_DEBUG_IRQFLAGS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_PI_LIST=y

[Arshad Hussain]

Alex, Thanks for reporting this. This is definitely refcount_t error. Could you please also put in the reproducer (steps/Test case) I will be happy to bisect this and look into this.

[Alex Zhuravlev]

not sure whether this is related, but hitting this very often:

[ 3112.751726] ------------[ cut here ]------------
[ 3112.752957] refcount_t hit zero at nrs_tbf_jobid_hop_put+0x10/0x20 [ptlrpc] in mdt00_005[9359], uid/euid: 0/0
[ 3112.753151] WARNING: CPU: 1 PID: 9359 at kernel/panic.c:703 refcount_error_report.cold.9+0x60/0x7b
[ 3112.753231] Modules linked in: zfs(O) zunicode(O) zzstd(O) zlua(O) zcommon(O) znvpair(O) zavl(O) icp(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O)
[ 3112.753574] CPU: 1 PID: 9359 Comm: mdt00_005 Tainted: G        W  O     --------- -  - 4.18.0 #11
[ 3112.753650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014
[ 3112.753726] RIP: 0010:refcount_error_report.cold.9+0x60/0x7b
[ 3112.753782] Code: 25 c0 ce 01 00 48 8b 95 80 00 00 00 48 8d 88 60 0b 00 00 48 c7 c7 88 f4 e1 a8 44 8b 80 20 09 00 00 56 48 89 de e8 21 fc ff ff <0f> 0b 58 e9 16 fc ff ff 8b 35 c1 73 f6 00 eb a8 44 8b 0d b8 73 f6
[ 3112.753940] RSP: 0018:ffffa327e0b83c30 EFLAGS: 00010286
[ 3112.753985] RAX: 0000000000000061 RBX: ffffffffa8e1dbb3 RCX: 0000000000000002
[ 3112.754049] RDX: 0000000000000002 RSI: ffffffffa8e39811 RDI: 00000000ffffffff
[ 3112.754114] RBP: ffffa327e0b83cc8 R08: 0000000000000000 R09: 0000000000000000
[ 3112.754179] R10: ffffa327e0b83a78 R11: ffffa327e0b83a70 R12: 0000000000000004
[ 3112.754244] R13: 0000000000000000 R14: ffffa327e0b83cc8 R15: ffffffffc06a365c
[ 3112.754309] FS:  0000000000000000(0000) GS:ffffa328b6d00000(0000) knlGS:0000000000000000
[ 3112.754574] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3112.754629] CR2: 000056317cd7cf44 CR3: 00000001d5050000 CR4: 0000000000350ea0
[ 3112.754696] Call Trace:
[ 3112.755488]  ex_handler_refcount+0x49/0x70
[ 3112.756600]  fixup_exception+0x2b/0x36
[ 3112.756681]  do_trap+0x45/0x110
[ 3112.756719]  do_invalid_op+0x23/0x30
[ 3112.756754]  ? ptlrpc_exit+0x90/0x114 [ptlrpc]
[ 3112.758640]  invalid_op+0x14/0x20
[ 3112.758705] RIP: 0010:nrs_tbf_jobid_hop_put+0x10/0x20 [ptlrpc]
[ 3112.759088] Code: 66 2e 0f 1f 84 00 00 00 00 00 f0 ff 46 54 0f 88 d8 7c 05 00 c3 0f 1f 44 00 00 f0 ff 4e 54 0f 84 ce 7c 05 00 0f 88 c8 7c 05 00 <c3> 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 55 48 89 fd 53 48
[ 3112.759238] RSP: 0018:ffffa327e0b83d70 EFLAGS: 00010246
[ 3112.759284] RAX: ffffffffc064bb80 RBX: ffffa328547dba00 RCX: ffffa32852a1e06c
[ 3112.759421] RDX: 0000000000000003 RSI: ffffa32852a1e018 RDI: ffffa328547dba00
[ 3112.761099] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 3112.761296] R10: ffffa32855dfd000 R11: ffffa32855dfc1be R12: ffffa32852a1e018
[ 3112.761493] R13: 0000000000000077 R14: ffffffffc00e9a78 R15: 0000000000000000
[ 3112.761560]  ? nrs_tbf_id_hop_get+0x10/0x10 [ptlrpc]
[ 3112.762286]  cfs_hash_putref+0x1ad/0x4d0 [libcfs]
[ 3112.762342]  nrs_tbf_stop+0x4a/0x2a0 [ptlrpc]
[ 3112.762514]  nrs_policy_started_put+0x35/0x1d0 [ptlrpc]
[ 3112.763111]  ptlrpc_nrs_req_finalize+0x1a/0x30 [ptlrpc]
[ 3112.764220]  ptlrpc_server_finish_active_request+0x48/0x120 [ptlrpc]
[ 3112.764359]  ptlrpc_main+0x13cf/0x3530 [ptlrpc]
[ 3112.764486]  ? _raw_spin_unlock_irqrestore+0x43/0x70
[ 3112.764536]  ? ptlrpc_wait_event+0x530/0x530 [ptlrpc]
[ 3112.765342]  kthread+0x16e/0x1a0
[ 3112.765688]  ? set_kthread_struct+0x40/0x40
[ 3112.838547]  ret_from_fork+0x24/0x30
[ 3112.838594] irq event stamp: 0
[ 3112.838628] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[ 3112.838686] hardirqs last disabled at (0): [<ffffffffa80ddd45>] copy_process+0x4c5/0x1bb0
[ 3112.838756] softirqs last  enabled at (0): [<ffffffffa80ddd45>] copy_process+0x4c5/0x1bb0
[ 3112.838893] softirqs last disabled at (0): [<0000000000000000>] 0x0
[ 3112.838960] ---[ end trace de302b6ab21f5dde ]---
[ 3112.854924] ------------[ cut here ]------------