[LU-12406] conf-sanity test 111 fails with ‘add mds1 failed with new params’

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Cannot Reproduce
• Severity: 3

Description

conf-sanity test_111 fails with ‘add mds1 failed with new params’. Looking at the client test_log, we see that the problem is with the --mkfsoptions flag in the mkfs command

CMD: trevis-18vm11 mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O large_dir -i 1048576 -O ea_inode -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey
trevis-18vm11: mkfs.lustre: don't specify multiple -O options
trevis-18vm11: 
trevis-18vm11: mkfs.lustre FATAL: mkfs failed 22
trevis-18vm11: mkfs.lustre: exiting with 22 (Invalid argument)

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: sys.timeout=20 mdt.identity_upcall=/usr/sbin/l_getidentity

device size = 2048MB
 conf-sanity test_111: @@@@@@ FAIL: add mds1 failed with new params

It looks like Lustre does not like having multiple ‘-O’ flags in the --mkfsoptions. This may be an issue with the test and forming the --mkfsoptions list.

Looking in the client test_log, we also see that test_115 suffers from this same issue, but the test is skipped due to this issue

== conf-sanity test 115: Access large xattr with inodes number over 2TB ============================== 02:17:13 (1559787433)
Stopping clients: trevis-18vm4.trevis.whamcloud.com,trevis-18vm5 /mnt/lustre (opts:)
CMD: trevis-18vm4.trevis.whamcloud.com,trevis-18vm5 running=\$(grep -c /mnt/lustre' ' /proc/mounts);
if [ \$running -ne 0 ] ; then
echo Stopping client \$(hostname) /mnt/lustre opts:;
…
CMD: trevis-18vm11 mkfs.lustre --mgsnode=trevis-18vm11@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O ea_inode -E lazy_itable_init\" --device-size=3298534883328 --mkfsoptions='-O lazy_itable_init,ea_inode,^resize_inode,meta_bg -i 1024' --mgs --reformat /dev/loop0
trevis-18vm11: 
trevis-18vm11: mkfs.lustre FATAL: Unable to build fs /dev/loop0 (256)
trevis-18vm11: 
trevis-18vm11: mkfs.lustre FATAL: mkfs failed 256

   Permanent disk data:
Target:     lustre:MDT0000
Index:      0
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x65
              (MDT MGS first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=10.9.4.221@tcp sys.timeout=20 mdt.identity_upcall=/usr/sbin/l_getidentity

device size = 3145728MB
formatting backing filesystem ldiskfs on /dev/loop0
	target name   lustre:MDT0000
	kilobytes     3221225472
	options        -i 1024 -J size=4096 -I 1024 -q -O lazy_itable_init,ea_inode,^resize_inode,meta_bg,dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -i 1024 -J size=4096 -I 1024 -q -O lazy_itable_init,ea_inode,^resize_inode,meta_bg,dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/loop0 3221225472k
   Invalid filesystem option set: lazy_itable_init,ea_inode,^resize_inode,meta_bg,dirdata,uninit_bg,^extents,dir_nlink,quota,huge_   file,flex_bg

 SKIP: conf-sanity test_115 format large MDT failed

It looks like this started failing on 2019-06-03 with Lustre 2.12.53.104. Logs for failed test sessions with this test failure are at
https://testing.whamcloud.com/test_sets/6b65b0f4-8695-11e9-869c-52540065bddc
https://testing.whamcloud.com/test_sets/34c0a1a0-8698-11e9-af1f-52540065bddc
https://testing.whamcloud.com/test_sets/efc1357e-8895-11e9-8c65-52540065bddc

Issue Links

• is related to: LU-12540 overstripe landing broke a MKFS optio...
• is related to: LU-11546 enable large_dir support for MDTs
• is related to: LU-14345 e2fsck of very large directories is b...

Comments

[Jian Yu]

+1 on master branch: https://testing.whamcloud.com/test_sets/496cea80-bf2a-11e9-a2b6-52540065bddc

[Andreas Dilger]

This seems very similar to LU-12540, which recently landed patch https://review.whamcloud.com/35482 "LU-12540 tests: squash multiply -O -E options".

I'm not totally satisfied with that solution, since it only fixes the test-framework to merge multiple options and not mkfs.lustre itself. That said, maybe a similar fix is needed for this test as well? I see this test failing a lot on Oleg's test hardware, so it would be nice to clear that up, even if the solution is sub-optimal.

[Andreas Dilger]

It looks like this problem is at least avoided by patch https://review.whamcloud.com/35358 "LU-11546 tests: enable large_dir support for tests", since that adds the "large_dir" option into the standard mkfs_opts(), so it isn't explicitly added to the end of the other options in test_111().

While that doesn't completely solve the problem (it would still be an error if large_dir was removed from fs_mkfs_opts, it would essentially never be hit again and this ticket could be closed as "Won't fix".

[Andreas Dilger]

Note that the 35482 patch doesn't fix this because it is removing duplicates inside mkfs_opts(), while test_111() is adding them afterward.

[Andreas Dilger]

conf-sanity test_111 passing regularly on master, when not skipped because of SLOW=no