[LU-11965] interop: sanity test 311 fails with 'create_count is zero'

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.13.0
• Fix Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: James Nunez (Inactive)
• Resolution: Fixed
• Labels: interop

Environment

2.10.6 servers with master (>= 2.12.50) clients and 2.12.0 servers with master clients

• Severity: 3

Description

sanity test_311 is failing with the error 'create_count is zero' in interop testing. Test 311 was recently modified by a patch for LU-11605; https://review.whamcloud.com/#/c/33559/

       unlinkmany $DIR/$tdir/$tfile. 1000
 
-       for idx in $(seq $MDSCOUNT); do
-               do_facet mds$idx "lctl set_param -n \
-                       osp.*OST0000*MDT000?.max_create_count=$max_count"
-               do_facet mds$idx "lctl set_param -n \
-                       osp.*OST0000*MDT000?.create_count=$count"
-       done
+       do_nodes $mdts "$LCTL set_param -n \
+                       osp.*OST0000*.max_create_count=$max_count"
+       [ $(lustre_version_code $facet) -lt $(version_code 2.11.56) ] &&
+               do_nodes $mdts "$LCTL set_param -n \
+                               osp.*OST0000*.create_count=$count"
+       do_nodes $mdts "$LCTL get_param osp.*OST0000*.create_count" |
+                       grep "=0" && error "create_count is zero"
 
        local new_iused
        for i in $(seq 120); do

There’s at least one issue with this patch; $facet is never defined for this test and, if test 311 is run alone, $facet is not defined. It looks like we are checking the server version of the client. We should be checking the Lustre version of mds1. This may not be the only issue with the test.

This test is failing for 2.10.6 servers with master (>= 2.12.50) clients and 2.12.0 servers with master clients. Logs for these failures are at:
https://testing.whamcloud.com/test_sets/bc9b2380-2ea4-11e9-9b3a-52540065bddc
https://testing.whamcloud.com/test_sets/fd4cd52a-2e56-11e9-9b3a-52540065bddc
https://testing.whamcloud.com/test_sets/0f4dfb86-2b38-11e9-90fb-52540065bddc
https://testing.whamcloud.com/test_sets/abd76ad6-292c-11e9-b901-52540065bddc
https://testing.whamcloud.com/test_sets/002d2f4c-26e6-11e9-8486-52540065bddc

Issue Links

• is related to: LU-11605 create_count stuck in 0 after changei...
• is related to: LU-11208 Interop 2.10.4<->master sanity test_2...
• is related to: LU-11607 Reduce repeated function calls in Lus...

Comments

[James Nunez (Inactive)]

Sergey -
Would you please look into this failure; I know that $facet needs to be defined/changed, but is there any other reason this test now fails in interop testing?

Thank you

[Andreas Dilger]

It might just need the use of "$MDS1_VERSION" instead of "$(lustre_version_code $facet)". $MDS1_VERSION is new for test scripts since v2_12_50-84-g4eb4479b0e.

[James Nunez (Inactive)]

Since this patch landed to master, I think the version check should be against 2.12.50.

[James Nunez (Inactive)]

I've modified the patch for LU-11208, https://review.whamcloud.com/#/c/33420/ , to fix this issue. Please review the patch and see if you agree with the fix.

[Sergey Cheremencev]

Hello James,
Agree with Andreas - $MDS1_VERSION should help with sanity_311.

[James Nunez (Inactive)]

Patch to add version check landed to 2.13.