[LU-15926] nrs_tbf.c:3080:nrs_tbf_req_get(): ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs ) failed

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Environment: Lustre 2.15.50 + VMs + ldiskfs
• Severity: 3
• Assignee: Etienne Aujames
• Reporter: Etienne Aujames

Description

Step to reproduce:

• Set a tbf rule with "realtime=1":

     lctl set_param mds.MDS.mdt.nrs_tbf_rule="start test_rule uid={10} rate=10 realtime=1"
  

• Generate rpc traffic matching the rule (superior to the tbf rate):

    while true; do printf "%s\n" test{001..100} | xargs -P4 -I{} bash -c "touch {}; rm -rf {}"; done
  

• Change the tbf rate:

   lctl set_param mds.MDS.mdt.nrs_tbf_rule="change test_rule rate=1"
  

• Crash

Calltrace

[23063.366296] LustreError: 28373:0:(qsd_reint.c:635:qqi_reint_delayed()) lustrefs-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[39267.608802] LustreError: 27261:0:(nrs_tbf.c:3080:nrs_tbf_req_get()) ASSERTION( cli->tc_nsecs_resid < cli->tc_nsecs ) failed: 
[39267.608846] LustreError: 27261:0:(nrs_tbf.c:3080:nrs_tbf_req_get()) LBUG
[39267.608869] Pid: 27261, comm: mdt00_000 3.10.0-1160.59.1.el7.centos.plus.x86_64 #1 SMP Wed Feb 23 17:40:21 UTC 2022
[39267.608870] Call Trace:
[39267.608887] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs]
[39267.608891] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs]
[39267.608955] [<0>] nrs_tbf_req_get+0x343/0x380 [ptlrpc]
[39267.609049] [<0>] ptlrpc_nrs_req_get_nolock0+0x75/0x170 [ptlrpc]
[39267.609078] [<0>] ptlrpc_server_request_get+0x1b7/0x280 [ptlrpc]
[39267.609112] [<0>] ptlrpc_server_handle_request+0x3f/0xc30 [ptlrpc]
[39267.609138] [<0>] ptlrpc_main+0xbf4/0x15e0 [ptlrpc]
[39267.609141] [<0>] kthread+0xd1/0xe0
[39267.609145] [<0>] ret_from_fork_nospec_begin+0x21/0x21
[39267.609160] [<0>] 0xfffffffffffffffe