[LU-17920] Add permanent TBF rules

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Labels: TBF
• Severity: 3
• Assignee: Etienne Aujames
• Reporter: Etienne Aujames

Description

Nowadays, this is not possible to add several permanents TBF rules via "lctl set_param -P".
The MGS will erase the old value of a parameter in the llog "CONFIG/params".

For example if we want 2 add the following rules:

# lctl set_param -P ost.OSS.ost_io.nrs_policies="tbf jobid"
# lctl set_param -P ost.OSS.ost_io.nrs_tbf_rule="start login jobid={*login*} rate=5000"
# lctl set_param -P ost.OSS.ost_io.nrs_tbf_rule="start rbh jobid={*rbh*} rate=100000"
# lctl get_param ost.OSS.ost_io.nrs_tbf_rule
ost.OSS.ost_io.nrs_tbf_rule=
regular_requests:
CPT 0:
rbh {*rbh*} 100000, ref 0
login {*login*} 5000, ref 0
...
# lctl llog_print params | grep tbf
- { index: 14, event: set_param, device: general, parameter: ost.OSS.ost_io.nrs_policies, value: tbf jobid }
- { index: 20, event: set_param, device: general, parameter: ost.OSS.ost_io.nrs_tbf_rule, value: start rbh jobid={*rbh*} rate=100000 }

Only the last one is set in the llog config.

So, if we unmount/remount the OST:

# lctl get_param ost.OSS.ost_io.nrs_tbf_rule
ost.OSS.ost_io.nrs_tbf_rule=
regular_requests:
CPT 0:
rbh {*rbh*} 100000, ref 0
default {*} 10000, ref 0

For now, at the CEA, we use udev rules to automatically set permanent tbf rules. But this is not really clean (hard sleep to wait for lustre startup etc...).

I see several way to fix that issue:

• The simplest way would be to add "--append" option to "set_param -P". This would advise the MGS to not remove the old entries. But this will not be easy for admin to maintain those entries: if he want to remove one rule at startup, he has to resent all the rules. Then, to overcome that issue, we could extend "lctl set_param -P -d" to match value too.
• We can also add a special cases to handle "set_param -P" for "nrs_tbf_rules". And we can manage this like for OST pools.
• We can also make all TBF rules permanent and add specific ldiskfs files to store those rules for each target (iam files ?). But this seems complex.

Issue Links

• is related to: LU-18929 Inconsistent Lustre parameters for new MDT/OST
• is related to: LU-13037 print stats for NRS TBF rules
• is related to: LU-17296 NRS TBF default rules
• is related to: LU-18183 shared rate limit for a TBF rule
• is related to: LU-9544 'lctl set_param -d fail_val=42' sets fail_val to 42
• is related to: LU-14501 NRS TBF UID: limit per "any" user?

Comments

[Peter Jones]

Merged for 2.17

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56218/
Subject: LU-17920 mgs: handle compound permanent parameters
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f69f27cc354741f797856ffb7c21474ebdab44c8

[Andreas Dilger]

Assign back to Etienne, as he already pushed a patch to address this.

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56218
Subject: LU-17920 mgs: handle compound permanent parameters
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ecf8e9d9b8c9c09bf8592fefed7804d92c5bcad8

[Etienne Aujames]

I have something in my stash for a little while, I will try to cleanup this today.
The approach is:

1. Allow multiple conf llog records for nrs_tbf_rule/pcc
2. Implement "lctl set_param -P -d <param>=<value>" to cancel only the llog records matching the parameter with the value

[Andreas Dilger]

The "lctl set_param PARAM=VALUE" command is not permanently setting a parameter, it only sets the parameter on the running system and this setting will be lost if the server restarts. The issue here is that "lctl set_param -P PARAM=VALUE" that sets parameters permanently in the configuration log (shown by "lctl llog_print params") will cancel an existing PARAM saved in the config log record and add a new one if PARAM is the same, since most parameters do not benefit from being set multiple times (last one wins).

So this ticket is about some way that it is possible for lctl set_param -P" to not cancel the previous record(s) for nrs_tbf_rule (and pcc which has the same issue).

[Chuck Ritter]

I think what you are saying is that you can "stack" rules if you set the policy afterward. Currently, we set the policy first and which ever rule is added last will clobber all other rules in the stack. Does this work at the current code level of Exa 6.3 or is it in your fix?

lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start other_clients nid=

{client_nid_ranges@o2ib1} rate=750"
lctl set_param mds.MDS.mdt.nrs_tbf_rule="start other_clients nid={client_nid_ranges@o2ib1}

rate=750"
lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start iozone_user jobid=

{name convention} rate=100”
lctl set_param mds.MDS.mdt.nrs_policies="tbf"
lctl set_param ost.OSS.ost_io.nrs_policies="tbf"'

instead of

lctl set_param mds.MDS.mdt.nrs_policies="tbf nid"
lctl set_param ost.OSS.ost_io.nrs_policies="tbf jobid"
lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start other_clients nid={client_nid_ranges@o2ib1} rate=750"
lctl set_param mds.MDS.mdt.nrs_tbf_rule="start other_clients nid={client_nid_ranges@o2ib1} rate=750"
lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start iozone_user jobid={name convention}

rate=100”

[Michael Aguilar]

note that I didn't specify the type of filter, until I deployed that filter inline

[Michael Aguilar]

lctl set_param mds.MDS.mdt.nrs_policies="tbf"

lctl set_param ost.OSS.ost_io.nrs_policies="tbf"'

[Michael Aguilar]

I have a running solution to this, as follows:

 

lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start other_clients nid={client_nid_ranges@o2ib1} rate=750"   lctl set_param mds.MDS.mdt.nrs_tbf_rule="start other_clients nid={client_nid_ranges@o2ib1} rate=750"   lctl set_param ost.OSS.ost_io.nrs_tbf_rule=start iozone_user jobid={name convention} rate=100”