[LU-15751] Interop conf-sanity test_32b: Fails list verification with missing permissions

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.15.0, Lustre 2.15.3
• Severity: 3
• Assignee: Sarah Liu
• Reporter: Maloo

Description

This issue was created by maloo for Cliff White <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/17d09693-cea1-4d75-a1ce-142830fe1ca2

test_32b failed with the following error:

list verification failed

Logs show:
-144115205289279497 -rw-r--r-- 1 0 0  473 1536307248 rc.local
-144115205289279495 -rwxr-xr-x 1 0 0  7923 1535106207 network
 144115205289279490 -rwxr-xr-x 1 0 0   868 1499979161 lsvcgss
+                 ? -????????? ? ? ?     ? ? network
+                 ? -????????? ? ? ?    ? ? rc.local
+                 ? -????????? ? ? ?     ? ? README
+                 ? -????????? ? ? ?     ? ? rhnsd
 conf-sanity test_32b: @@@@@@ FAIL: list verification faile

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
conf-sanity test_32b - list verification failed

Issue Links

• is related to: LU-15506 conf-sanity test_32b failed with 2: sha1sum verification failed
• is related to: LU-14853 conf-sanity: create ldiskfs and zfs filesystem upgrade images for 2.15
• is related to: LU-11643 create disk images for Lustre 2.10 and 2.12 for ldiskfs

Comments

[Andreas Dilger]

It looks like this is only failing for interop test cases with 2.12.x client, master server, and fails in the same way for both the 2.10 and 2.12 disk images. It looks like the problem is because the ost2 image is no mounted by test_32b in 2.12, and the "stat" on these OST objects is failing, as can be seen in the client console logs:

lsme_unpack()) t32fs-clilov_UUID: OST index 1 more than OST count 1
lov_dump_lmm_common()) objid 0x9:1026, magic 0x0bd10bd0, pattern 0x1
lov_dump_lmm_common()) stripe_size 1048576, stripe_count 1, layout_gen 0
lov_dump_lmm_objects()) stripe 0 idx 1 subobj 0x0:5
cl_file_inode_init()) t32fs: failed to initialize cl_object [0x200000402:0x9:0x0]: rc = -22
ll_prep_inode()) new_inode -fatal: rc -22
lov_dump_lmm_common()) objid 0x9:1026, magic 0x0bd10bd0, pattern 0x1
lov_dump_lmm_common()) stripe_size 1048576, stripe_count 1, layout_gen 0
lov_dump_lmm_objects()) stripe 0 idx 1 subobj 0x0:5

and the missing inode number 144115205289279497=0x200000402000009 matches the FID reported by cl_file_inode_init().

Even though the new 2.10 and 2.12 images are coming from the MDS (master) installation, the conf-sanity test_32b script runs from b2_12, so it will not have the LU-15506 test_32b fixes. This looks like a test script interop issue. Probably the best thing to do is backport the LU-11643 and LU-15506 patches to b2_12 for the 2.12.9 release so that they have the test_32b script fixes and the images.

In the future, it makes sense to have the script loop over all of the MDT and OST images for processing and mounting, instead of explicitly checking for mdt and mdt2 and ost and ost2. That would avoid future problems if there are further MDT or OST images added to the tarballs.

[James Nunez (Inactive)]

Sarah -
Would you please take a look at this issue?

Thank you

[James Nunez (Inactive)]

Conf-sanity test 32b started to fail with the 'list verification' error when the patch adding the b2_10 and b2_12 images to be upgraded from. The first time we see this test fail is for b2_12 image on January 28, 2022 at https://testing.whamcloud.com/test_sets/17d09693-cea1-4d75-a1ce-142830fe1ca2.