[LU-18439] LLTPfcntl tests are failing during regression IO test

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.16.0

Environment

79 x Lustre 2.16 RC clients

4 X LNet routers

Lustre client:

2.16.0_RC5_4_g60e267b

NEO version: 6.7-020.141

Lustre server:

2.16.0_RC5_4_g60e267b

Model SSUs

1xE1000D 1xE1000F

4 x OSS nodes with 2 x HDD OSTs and 2 x flash OSTs

• Severity: 3
• Assignee: Rajeev Mishra
• Reporter: Prasannakumar Nagasubramani

Description

Bunch of CL_LLTP/LLTPfcntl tags failed during regression IO run. Found these kind of failures on tag output.

 Job Script: command started at Wed Oct 16 01:52:35 CDT 2024
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 1
 fcntl19     0  [1;34mTINFO[0m  :  Test block 1: PASSED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 1
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 2
 fcntl19     0  [1;34mTINFO[0m  :  Test block 2: PASSED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 2
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 3
 fcntl19     0  [1;34mTINFO[0m  :  Test block 3: PASSED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 3
 fcntl19     0  [1;34mTINFO[0m  :  Enter blcok 4
 fcntl19     0  [1;34mTINFO[0m  :  Test block 4: PASSED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 4
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 5
 fcntl19     1  [1;31mTFAIL[0m  :  fcntl19.c:190: region length is wrong, should be 3 is 7
 fcntl19     0  [1;34mTINFO[0m  :  Test block 5: FAILED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 5
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 6
 fcntl19     2  [1;31mTFAIL[0m  :  fcntl19.c:190: region length is wrong, should be 4 is 8
 fcntl19     0  [1;34mTINFO[0m  :  Test block 6: FAILED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 6
 fcntl19     0  [1;34mTINFO[0m  :  Enter block 7
 fcntl19     0  [1;34mTINFO[0m  :  Test block 7: PASSED
 fcntl19     0  [1;34mTINFO[0m  :  Exit block 7
 Application 8246098 exit codes: 1 

Test command line : 

test cmdline ......: ubrun -t -o -T LLTPfcntl11 -E "LTPROOT=$LLTPROOT" -e LLTPROOT -p LLTPROOT=testcases/bin LLTPROOT=testcases/bin/fcntl11 

Passed job :

ubrun: Env LLTPROOT_CL=/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/xtcnl/ltp.latest/ROOT.latest
ubrun: Prepending path  : '/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/xtcnl/ltp.latest/ROOT.latest/testcases/bin'
ubrun: Binary Info: -rwxrwxr-x 1 vers tsttool 392704 May  7  2021 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/xtcnl/ltp.latest/ROOT.latest/testcases/bin/ftest04
ubrun: Execute Cmd: '  aptrun -n1 -M1 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/xtcnl/ltp.latest/ROOT.latest/testcases/bin/ftest04'
CL_LLTPftest04    1  PASS  :  No failures found with the command 'aptrun'     + The return value was 0 as expected.
aptrun : Launch cmd is 'aprun -n 1 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/xtcnl/ltp.latest/ROOT.latest/testcases/bin/ftest04'
Job Script: command started at Sun Nov 3 23:18:35 CST 2024
ftest04     1  TPASS  :  Test passed.
Application 8328916 resources: utime ~0s, stime ~1s, Rss ~9100, inblocks ~2152, outblocks ~83832
Job Script: command stopped at Sun Nov 3 23:18:39 CST 2024
Job Script: command runtime was 4 seconds

jupiter-p1:/lus/kjcf05/nivi/rerun # ubrun -t -o -T LLTPmsgctl11 -E "LTPROOT=$LLTPROOT" -e LLTPROOT -p LLTPROOT=testcases/bin LLTPROOT=testcases/bin/msgctl11
ubrun: Env LLTPROOT=/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest
ubrun: Prepending path  : '/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin'
ubrun: Binary Info: -rwxrwxr-x 1 vers tsttool 414048 May  7  2021 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/msgctl11
ubrun: Execute Cmd: '  /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/msgctl11'
LLTPmsgctl11    1  PASS  :  No failures found with the command 'LLTPROOT=testcases/bin/msgctl11'     + The return value was 0 as expected.
msgctl11    0  TINFO  :  Found 32000 available message queues
msgctl11    0  TINFO  :  Using upto 16114 pids
msgctl11    1  TPASS  :  msgctl11 ran successfully!
jupiter-p1:/lus/kjcf05/nivi/rerun #

jupiter-p1:/lus/kjcf05/nivi/rerun #  jupiter-p1:/lus/kjcf05/nivi/rerun # ubrun -t -o -T LLTPftest01 -E "LTPROOT=$LLTPROOT" -e LLTPROOT -p LLTPROOT=testcases/bin LLTPROOT=testcases/bin/ftest01
ubrun: Env LLTPROOT=/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest
ubrun: Prepending path  : '/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin'
ubrun: Binary Info: -rwxrwxr-x 1 vers tsttool 400080 May  7  2021 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/ftest01
ubrun: Execute Cmd: '  /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/ftest01'
LLTPftest01    1  PASS  :  No failures found with the command 'LLTPROOT=testcases/bin/ftest01'     + The return value was 0 as expected.
ftest01     1  TPASS  :  Test passed in fork and wait.
ftest01     2  TPASS  :  Test passed.

jupiter-p1:/lus/kjcf05/nivi/rerun # ubrun -t -o -T LLTPftest06 -E "LTPROOT=$LLTPROOT" -e LLTPROOT -p LLTPROOT=testcases/bin LLTPROOT=testcases/bin/ftest06
ubrun: Env LLTPROOT=/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest
ubrun: Prepending path  : '/hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin'
ubrun: Binary Info: -rwxrwxr-x 1 vers tsttool 393168 May  7  2021 /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/ftest06
ubrun: Execute Cmd: '  /hpcdc/project/craytest/ostest/binaries/xt/rel.70up03.aries.cray-sp2/baselinux/ltp.latest/ROOT.latest/testcases/bin/ftest06'
LLTPftest06    1  PASS  :  No failures found with the command 'LLTPROOT=testcases/bin/ftest06'     + The return value was 0 as expected.
ftest06     1  TPASS  :  Test passed.
ftest06     2  TPASS  :  Test passed.
jupiter-p1:/lus/kjcf05/nivi/rerun #  

console-20241103

Issue Links

• duplicates: LU-17276 Use interval tree in flock to improve the scalability