[LU-11965] interop: sanity test 311 fails with 'create_count is zero'

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.13.0
• Labels: interop
• Environment: 2.10.6 servers with master (>= 2.12.50) clients and 2.12.0 servers with master clients
• Severity: 3
• Assignee: James Nunez (Inactive)
• Reporter: James Nunez (Inactive)

Description

sanity test_311 is failing with the error 'create_count is zero' in interop testing. Test 311 was recently modified by a patch for LU-11605; https://review.whamcloud.com/#/c/33559/

       unlinkmany $DIR/$tdir/$tfile. 1000
 
-       for idx in $(seq $MDSCOUNT); do
-               do_facet mds$idx "lctl set_param -n \
-                       osp.*OST0000*MDT000?.max_create_count=$max_count"
-               do_facet mds$idx "lctl set_param -n \
-                       osp.*OST0000*MDT000?.create_count=$count"
-       done
+       do_nodes $mdts "$LCTL set_param -n \
+                       osp.*OST0000*.max_create_count=$max_count"
+       [ $(lustre_version_code $facet) -lt $(version_code 2.11.56) ] &&
+               do_nodes $mdts "$LCTL set_param -n \
+                               osp.*OST0000*.create_count=$count"
+       do_nodes $mdts "$LCTL get_param osp.*OST0000*.create_count" |
+                       grep "=0" && error "create_count is zero"
 
        local new_iused
        for i in $(seq 120); do

There’s at least one issue with this patch; $facet is never defined for this test and, if test 311 is run alone, $facet is not defined. It looks like we are checking the server version of the client. We should be checking the Lustre version of mds1. This may not be the only issue with the test.

This test is failing for 2.10.6 servers with master (>= 2.12.50) clients and 2.12.0 servers with master clients. Logs for these failures are at:
https://testing.whamcloud.com/test_sets/bc9b2380-2ea4-11e9-9b3a-52540065bddc
https://testing.whamcloud.com/test_sets/fd4cd52a-2e56-11e9-9b3a-52540065bddc
https://testing.whamcloud.com/test_sets/0f4dfb86-2b38-11e9-90fb-52540065bddc
https://testing.whamcloud.com/test_sets/abd76ad6-292c-11e9-b901-52540065bddc
https://testing.whamcloud.com/test_sets/002d2f4c-26e6-11e9-8486-52540065bddc

Issue Links

• is related to: LU-11605 create_count stuck in 0 after changeing max_create_count to 0 and back 20 000
• is related to: LU-11208 Interop 2.10.4<->master sanity test_27G: 'testpool' is not empty
• is related to: LU-11607 Reduce repeated function calls in Lustre test suites

Comments

[James Nunez (Inactive)]

Patch to add version check landed to 2.13.

[Sergey Cheremencev]

Hello James,
Agree with Andreas - $MDS1_VERSION should help with sanity_311.

[James Nunez (Inactive)]

I've modified the patch for LU-11208, https://review.whamcloud.com/#/c/33420/ , to fix this issue. Please review the patch and see if you agree with the fix.

[James Nunez (Inactive)]

Since this patch landed to master, I think the version check should be against 2.12.50.

[Andreas Dilger]

It might just need the use of "$MDS1_VERSION" instead of "$(lustre_version_code $facet)". $MDS1_VERSION is new for test scripts since v2_12_50-84-g4eb4479b0e.

[James Nunez (Inactive)]

Sergey -
Would you please look into this failure; I know that $facet needs to be defined/changed, but is there any other reason this test now fails in interop testing?

Thank you