[LU-8008] Can't enable or add rules to TBF

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Major
• Reporter: Mahmoud Hanafi
• Assignee: Emoly Liu
• Resolution: Won't Fix
• Severity: 3

Description

I am trying to enable NRS_TBF but getting errors.

nbp1-oss6 ~ # cat /proc/fs/lustre/version 
lustre: 2.7.1
kernel: 2.6.32-504.30.3.el6.20151008.x86_64.lustre271
build:  2.7.1-2nasS_mofed31v5

nbp1-oss6 ~ # lctl set_param ost.OSS.ost_io.nrs_policies=tbf
ost.OSS.ost_io.nrs_policies=tbf
error: set_param: setting /proc/fs/lustre/ost/OSS/ost_io/nrs_policies=tbf: Invalid argument

nbp1-oss6 ~ # lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start rule1  { 10.151.27.22@2oib } 50"
ost.OSS.ost_io.nrs_tbf_rule=start rule1  { 10.151.27.22@2oib } 50
error: set_param: setting /proc/fs/lustre/ost/OSS/ost_io/nrs_tbf_rule=start rule1  { 10.151.27.22@2oib } 50: Invalid argument

Issue Links

• is related to: LU-7470 Extend TBF policy with NID/JobID expr...
• is related to: LU-8006 Specify ordering of TBF policy rules
• is related to: LU-3558 NRS TBF policy for QoS purposes

Comments

[Emoly Liu]

Here is some introduction on NRS-TBF for your reference:

3.1 Enable TBF policy

The command to enable a TBF policy is shown as follows:
#lctl set_param x.x.x.nrs_policies=”tbf [reg|hp] <type>”

The argument "type" is the classification type of RPC requests. Currently, only "nid" and "jobid" are supported.
For example,
#lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"
#lctl set_param ost.OSS.ost_io.nrs_policies="tbf reg nid"
#lctl set_param ost.OSS.ost_io.nrs_policies="tbf hp nid"

3.2 Start rule

The command to start a TBF policy rule is shown as follows:
#lctl set_param x.x.x.nrs_policies=”[reg|hp] start <rule_name> <arguments>...”

The argument "rule_name" argument is the TBF policy rule name. The format of the ‘arguments’ is different per the different TBF policy type.

For NID based TBF policy, the format is shown as follows:
#lctl set_param x.x.x.nrs_policies=”[reg|hp] start <rule_name> {<nid_list>} <rate>”

The ‘nid_list’ argument uses the same format used to configure an LNET route. The ‘rate’ argument is the RPC rate of the rule.

For example, 
#lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start loginnode {192.168.1.1@tcp} 10000"
#lctl set_param ost.OSS.ost_io.nrs_tbf_rule="reg start loginnode {192.168.1.1@tcp} 10000"
#lctl set_param ost.OSS.ost_io.nrs_tbf_rule="hp start loginnode {192.168.1.1@tcp} 10000"
#lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start computenodes {192.168.1.[1-128]@tcp} 1000"
#lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start other_clients {192.168.*.*@tcp} 100"

Here is the output of your example with correct format, on my computer:

[root@centos6-1 tests]# ../utils/lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"
ost.OSS.ost_io.nrs_policies=tbf nid
[root@centos6-1 tests]# ../utils/lctl set_param ost.OSS.ost_io.nrs_tbf_rule="start rule1 {10.151.27.22@o2ib} 50"
ost.OSS.ost_io.nrs_tbf_rule=start rule1 {10.151.27.22@o2ib} 50

You can find the similar introduction in Lustre manual, or the wiki page https://wiki.hpdd.intel.com/display/ENG/Token+Bucket+Filter+Policy.

[Mahmoud Hanafi]

That link doesn't work.

Is the type (nid or jobid) mutually exclusive? Or can we use both at the same time.

[Emoly Liu]

The work to use both at the same time is in progress in LU-7470 (http://review.whamcloud.com/#/c/17345/).

BTW, I converted that wiki page into pdf format and attached here, but some animation gif images can't be shown correctly.

[Jay Lan (Inactive)]

Hmmm, http://review.whamcloud.com/#/c/17345/ marked it conflicts with LU-8006, another ticket Mahmoud opened.

So, we can only have one solution?

[Emoly Liu]

I think we can discuss this issue with LiXi in LU-8006 or LU-7470 directly.

[John Fuchs-Chesney (Inactive)]

Resolving this because the work is being done on LU-8006.

~ jfc.