[LU-6921] sanityn 77f test failed Lustre: DEBUG MARKER: sanityn test_77f: @@@@@@ FAIL: failed to operate on TBF rules

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.8.0
• Labels: patch
• Severity: 3

Epic

test

• Assignee: WC Triage
• Reporter: Vinayak (Inactive)

Description

stdout.log

== sanityn test 77f: check TBF JobID nrs policy == 15:43:56 (1438011836)
ost.OSS.ost_io.nrs_policies=tbf jobid
ost.OSS.ost_io.nrs_policies=tbf jobid
error: set_param: ost/OSS/ost_io/nrs_tbf_rule: Found no match
 sanityn test_77f: @@@@@@ FAIL: failed to operate on TBF rules 
  Trace dump:
  = /usr/lib64/lustre/tests/../tests/test-framework.sh:4732:error_noexit()
  = /usr/lib64/lustre/tests/../tests/test-framework.sh:4763:error()
  = /usr/lib64/lustre/tests/sanityn.sh:2943:tbf_rule_operate()
  = /usr/lib64/lustre/tests/sanityn.sh:3008:test_77f()
  = /usr/lib64/lustre/tests/../tests/test-framework.sh:5010:run_one()
  = /usr/lib64/lustre/tests/../tests/test-framework.sh:5047:run_one_logged()
  = /usr/lib64/lustre/tests/../tests/test-framework.sh:4864:run_test()
  = /usr/lib64/lustre/tests/sanityn.sh:3043:main()
Dumping lctl log to /tmp/test_logs/1438011830/sanityn.test_77f.*.1438011837.log
FAIL 77f (1s)
sanityn: FAIL: test_77f failed to operate on TBF rules
Stopping clients: fre0107,fre0108 /mnt/lustre2 (opts:)
Stopping client fre0108 /mnt/lustre2 opts:

Issue Links

• is related to: LU-6939 nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli->tc_rule == ((void *)0) ) failed
• is related to: LU-7044 Interop 2.5.3<->master sanityn test_77e: failed to set TBF policy
• is related to: LU-6668 Add tests for TBF

Comments

[Andreas Dilger]

I finally figured out why this test wasn't failing in our testing - in facet_host() it uses $ost_HOST for any facet named ostX if there isn't an explicit $ost0_HOST set in the configuration.

In any case, the patch has been landed to master for 2.8.0.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/16305/
Subject: LU-6921 test: failed to operate on TBF rules
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: abbef8759e93c31a2c88ba650a04ae9076600afa

[Kalpak Shah (Inactive)]

http://review.whamcloud.com/#/c/16305/ is ready to be merged - Andreas and Li have given positive reviews.

[Vinayak (Inactive)]

Hello Suarabh,

Can you please try the patch http://review.whamcloud.com/#/c/16305/ and check if it fixes the problem.

Thanks,

[Saurabh Tandan (Inactive)]

Encountered same issue for sanity test_77g.

20:26:40:CMD: onyx-38vm4 lctl set_param ost.OSS.ost_io.nrs_tbf_rule=start\ dd_runas\ {dd.500}\ 50
20:26:40:onyx-38vm4: error: set_param: setting /proc/fs/lustre/ost/OSS/ost_io/nrs_tbf_rule=start dd_runas {dd.500} 50: Invalid argument
20:26:41:ost.OSS.ost_io.nrs_tbf_rule=start dd_runas {dd.500} 50
20:26:42: sanityn test_77g: @@@@@@ FAIL: failed to operate on TBF rules 

[Vinayak (Inactive)]

Hello Andreas,

I have rebased the patch.

http://review.whamcloud.com/#/c/16305/. Please let me know if anything else to be done.

[Vinayak (Inactive)]

Can anyone please let me know what is the behavior of this part (type) of script in sanityn.sh, test_77e, 77f, 77g

        tbf_rule_operate ost0 "start\ localhost\ {0@lo}\ 1000"

It is failing with
"error: set_param: ost/OSS/ost_io/nrs_tbf_rule: Found no match" on my local set up.

Is this behavior same on your side also ? Looks like ost0 is not correctly interpreted on my side.

If passes,
do we need to specify anything explicitly to make test work for ost0 (any changes to frame work or any environment variable need to set etc).

I am using 4 node set up (2 OSTs, 1 MDS, 2 clients)

[Gerrit Updater]

Vinayak ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16305
Subject: LU-6921 test: failed to operate on TBF rules
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e9064c1edc56a21a7687002894d044a3f8f9f1c5

[Vinayak (Inactive)]

Hi Li Xi
>> which branch did you test?
latest master

>> Was it a branch with TBF NRS policy?
Do we know if complete TBF NFS policy is present for sure in master? I don't know how can we find that out.

Followed this to find TBF related changes.

[root@cli-1 lustre-release]# git branch -a | grep master
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master

[root@cli-1 lustre-release]# git log --oneline | grep -i TBF
fb14b7b LU-6668 test: regression tests for NRS TBF policy
e7ab554 LU-5580 ptlrpc: policy switch directly in tbf
75752e9 LU-3319 procfs: Move NRS TBF proc handling to seq_files
0539dc5 LU-4832 ptlrpc: fix incorrect name string in nrs_tbf
33e35c0 LU-3558 ptlrpc: Add the NRS TBF policy

Please let me know if you want any other info or anything you want me to check on my side.
and also please correct me If I am missing any thing.

[Li Xi (Inactive)]

Hi Vinayak, which branch did you test? Was it a branch with TBF NRS policy?