[LU-17158] TBF rate should not be based on CPT Created: 29/Sep/23 Updated: 29/Sep/23 |
|
| Status: | Open |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor |
| Reporter: | Etienne Aujames | Assignee: | Etienne Aujames |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | tbf | ||
| Epic/Theme: | QoS-TBF |
| Severity: | 3 |
| Rank (Obsolete): | 9223372036854775807 |
| Description |
|
The TBF rate is related to Lustre CPU partition, that mean that the same TBF class can exist on several CPTs. So the maximum rate with several clients is: rate * nbr_cpt e.g: lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start rule_fio jobid={*} rate=3000
The job rate with 1 node will be limited 3000 RPC/s. But if the OSS is configured with 4 CPT and the job uses more than 4 nodes (or LNet routers when compute nodes are behind), the maximum rate is : I think this was originally design to keep the CPT independents and avoid lock contentions. But this can be mitigated by using rhashtable with rcu_lock to share tokens between CPT. |
| Comments |
| Comment by Andreas Dilger [ 29/Sep/23 ] |
|
As a starting point it would make sense to divide the rate by the CPT count so that the total rate remains the same regardless of how many CPTs are configured. Being able to share tokens between CPTs is an improvement beyond that. |