[LU-15926] nrs_tbf.c:3080:nrs_tbf_req_get(): ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs ) failed

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.16.0
• Type: Bug
• Priority: Major
• Reporter: Etienne Aujames
• Assignee: Etienne Aujames
• Resolution: Fixed

Environment

Lustre 2.15.50 + VMs + ldiskfs

• Severity: 3

Description

Step to reproduce:

• Set a tbf rule with "realtime=1":

     lctl set_param mds.MDS.mdt.nrs_tbf_rule="start test_rule uid={10} rate=10 realtime=1"
  

• Generate rpc traffic matching the rule (superior to the tbf rate):

    while true; do printf "%s\n" test{001..100} | xargs -P4 -I{} bash -c "touch {}; rm -rf {}"; done
  

• Change the tbf rate:

   lctl set_param mds.MDS.mdt.nrs_tbf_rule="change test_rule rate=1"
  

• Crash

Calltrace

[23063.366296] LustreError: 28373:0:(qsd_reint.c:635:qqi_reint_delayed()) lustrefs-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[39267.608802] LustreError: 27261:0:(nrs_tbf.c:3080:nrs_tbf_req_get()) ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs ) failed: 
[39267.608846] LustreError: 27261:0:(nrs_tbf.c:3080:nrs_tbf_req_get()) LBUG
[39267.608869] Pid: 27261, comm: mdt00_000 3.10.0-1160.59.1.el7.centos.plus.x86_64 #1 SMP Wed Feb 23 17:40:21 UTC 2022
[39267.608870] Call Trace:
[39267.608887] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs]
[39267.608891] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs]
[39267.608955] [<0>] nrs_tbf_req_get+0x343/0x380 [ptlrpc]
[39267.609049] [<0>] ptlrpc_nrs_req_get_nolock0+0x75/0x170 [ptlrpc]
[39267.609078] [<0>] ptlrpc_server_request_get+0x1b7/0x280 [ptlrpc]
[39267.609112] [<0>] ptlrpc_server_handle_request+0x3f/0xc30 [ptlrpc]
[39267.609138] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc]
[39267.609141] [<0>] kthread+0xd1/0xe0
[39267.609145] [<0>] ret_from_fork_nospec_begin+0x21/0x21
[39267.609160] [<0>] 0xfffffffffffffffe

Comments

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47585
Subject: LU-15926 nrs: fix tbf realtime rules
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 39e9ea421ff798d7a3e45dade4b0a616c2abb0dc

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47585/
Subject: LU-15926 nrs: fix tbf realtime rules
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 530861b344e46bef51c80adac4640c4586d8463a

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49467
Subject: LU-15926 nrs: fix tbf realtime rules
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: a422ca2b68977829c849b26975c6c0148d5a883e

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51364
Subject: LU-15926 nrs: fix tbf realtime rules
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 5baab1489ecbae0d951bbc86f20cf51238ada145