[LU-10216] One question about enabling NRS_TBF in Lustre

• Type: Question/Request
• Resolution: Fixed
• Priority: Critical
• Environment: Lustre 2.10.0/ Centos 7.3
• Assignee: Emoly Liu
• Reporter: sebg-crd-pm (Inactive)

Description

Hi All,

There are 4 X OSS servers and 2 X MGS/MDS in our Lustre filesystem. 

We are trying to enable NRS_TBF by using following instructions in our MGS server.

After this, we want to verify if these rules are added successfully by using following instructions in one OSS server. The result is shown below.

 

Is it correct?  If no, please kindly provide us with instructions to do this.

Also, how can we verify if these rules are "Change" and "Stop(Delete)" successfully when we use "lctl set_param ost.OSS.ost_io.nrs_tbf_rule="change ossb1 500" and "lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop ossb1", respectively.  

 

Thanks!

Issue Links

• is related to: LUDOC-328 documentation updates for complex TBF policies

Comments

[Emoly Liu]

Thanks.

[sebg-crd-pm (Inactive)]

You can close it. Thanks.

[Emoly Liu]

sebg-crd-pm,
Can we close this issue ?
Thanks,
Emoly

[Andreas Dilger]

Please note that "lctl set_param" commands are local to the server they are run on, and only in effect while the server is mounted, so these commands need to be run on all of the OSS nodes.

It looks like there is an open ticket for the NRS TBF documentation to be updated for the new functionality added by DDN - LUDOC-328.

[Emoly Liu]

Hi sebg-crd-pm,

The TBF policy format has been changed since lustre 2.8.55, as follows

• For NRS "tbf nid" policy:
  lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start $RULE_NAME nid={$NID} rate=$RATE"
• For NRS "tbf jobid" policy:
  lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start $RULE_NAME jobid={$JOBID} rate=$RATE"
• To change the policy:
  lctl set_param ost.OSS.ost_io.nrs_tbf_rule="change $RULE_NAME rate=$RATE"

So you should change your commands format to make them work. Here is my output for your reference:

[root@centos7-3 tests]# lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"
ost.OSS.ost_io.nrs_policies=tbf nid

[root@centos7-3 tests]# lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start ossb1 nid={172.20.110.209@o2ib} rate=100"
ost.OSS.ost_io.nrs_tbf_rule=start ossb1 nid={172.20.110.209@o2ib} rate=100
[root@centos7-3 tests]# lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start robin nid={172.20.110.212@o2ib} rate=200"
ost.OSS.ost_io.nrs_tbf_rule=start robin nid={172.20.110.212@o2ib} rate=200

[root@centos7-3 tests]# lctl get_param ost.OSS.ost_io.nrs_tbf_rule
ost.OSS.ost_io.nrs_tbf_rule=
regular_requests:
CPT 0:
ossb1 {172.20.110.209@o2ib} 100, ref 0
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0
high_priority_requests:
CPT 0:
ossb1 {172.20.110.209@o2ib} 100, ref 0
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0

[root@centos7-3 tests]# lctl set_param ost.OSS.ost_io.nrs_tbf_rule="change ossb1 rate=500"
ost.OSS.ost_io.nrs_tbf_rule=change ossb1 rate=500
[root@centos7-3 tests]# lctl get_param ost.OSS.ost_io.nrs_tbf_rule
ost.OSS.ost_io.nrs_tbf_rule=
regular_requests:
CPT 0:
ossb1 {172.20.110.209@o2ib} 500, ref 0
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0
high_priority_requests:
CPT 0:
ossb1 {172.20.110.209@o2ib} 500, ref 0
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0

[root@centos7-3 tests]# lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop ossb1"
ost.OSS.ost_io.nrs_tbf_rule=stop ossb1
[root@centos7-3 tests]# lctl get_param ost.OSS.ost_io.nrs_tbf_rule
ost.OSS.ost_io.nrs_tbf_rule=
regular_requests:
CPT 0:
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0
high_priority_requests:
CPT 0:
robin {172.20.110.212@o2ib} 200, ref 0
default {*} 10000, ref 0

HTH.
Emoly

[sebg-crd-pm (Inactive)]

Hi Emoly, 

Please give us some suggestions for this issue.

We really need your help.

Thanks!

 

[Peter Jones]

Emoly

Can you please advise?

Thanks

Peter