[LU-16861] Janitor Testing Fails to copy latest obdfilter-survey (Uses old obdfilter-survey)

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Environment: Client: 4.18.0-372.9.1.el8(8.5) ; Server: 4.18.0-425.3.1.el8(8.7)
• Severity: 3
• Assignee: Arshad Hussain
• Reporter: Arshad Hussain

Description

Testing/Fixing LU-16827 it was observerd that the run was failing in janitor testng. However it was passing in maloo. It was noticed that the required changes under lustre-iokit/obdfilter-survey/obdfilter-survey was not getting reflected under janitor. This was leading to janitor always failing while maloo run was passing as the latest (modified) script was getting used. It looks like janitor always used old script (at least for obdfilter-survey)

Here are the few revelant logs. Let me know if more information is requied. All logs are under https://review.whamcloud.com/c/fs/lustre-release/+/51035)

CASE 1: This is modfied to use specific path for obdfilter-survey to make janitor passing.

From logs : https://testing.whamcloud.com/gerrit-janitor/31629/testresults/obdfilter-survey-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/obdfilter-survey.test_1a.test_log.oleg368-client.log

This is the obdfilter-survey janitor always uses

ls -ali /usr/bin/obdfilter-survey
920857 -rwxr-xr-x. 1 root root 16279 Jun 4 2016 /usr/bin/obdfilter-survey

This is the obdfilter-survey which it is supposed to used. (There are changes withing this code -which maloo correctly picks it up). Please notice the binary size and date stamp for both.

 
ls -ali /home/green/git/lustre-release/lustre/../lustre-iokit/obdfilter-survey
total 42
...
2278 -rwxr-xr-x 1 green green 15632 May 31 07:47 obdfilter-survey
...

 

test_1a under obdfilter-survey.sh was modifed with to use specfic  OBDSURVEY instead of generic system path version (which is /usr/bin/obdfilter-survey). When this is done the test passes janitor. Else it fails.

export PATH=$PATH:/home/green/git/lustre-release/lustre/../lustre-iokit/obdfilter-survey
OBDSURVEY=/home/green/git/lustre-release/lustre/../lustre-iokit/obdfilter-survey/obdfilter-survey
obdflter_survey_run disk

CASE 2: The fail case

This uses old /usr/bin/obdfilter-survey and new changes are not reflected.

 

<snip>
+ eval NETTYPE=tcp thrlo=2 nobjhi=1 thrhi=4 size=1024 case=disk rslt_loc=/tmp 'targets="192.168.203.104:lustre-OST0000' '192.168.203.104:lustre-OST0001"' /usr/bin/obdfilter-survey
++ NETTYPE=tcp 
++ thrlo=2 
++ nobjhi=1 
++ thrhi=4 
++ size=1024 
++ case=disk 
++ rslt_loc=/tmp
++ targets='192.168.203.104:lustre-OST0000 192.168.203.104:lustre-OST0001' 
++ /usr/bin/obdfilter-survey
Warning: Permanently added '192.168.203.104' (ECDSA) to the list of known hosts. 
bash: lctl: command not found 
/usr/bin/obdfilter-survey: line 242: ( << 16) | ( << 8) | : syntax error: operand expected (error token is "<< 16) | ( << 8) | ") 
/usr/bin/obdfilter-survey: line 254: [: -lt: unary operator expected 
bash: lctl: command not found 
bash: lctl: command not found 
OST lustre-OST0000 not setup
<snip>

 

Issue Links

• has to be done before: LU-16827 obdfilter-survey: /usr/bin/obdfilter-survey failed: 2