Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1741] Test failure on test suite conf-sanity test_18 https://jira.whamcloud.com/browse/LU-1741 Lustre <p>This issue was created by maloo for Peter Jones <pjones@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/6379bf88-e46d-11e1-af05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6379bf88-e46d-11e1-af05-52540035b04c</a>.</p> <p>Error: 'expected journal size > 32M, found 32M' </p> LU-1741

Test failure on test suite conf-sanity test_18

Bug Blocker Resolved Fixed Zhenyu Xu Maloo Sun, 12 Aug 2012 11:05:19 +0000 Thu, 18 Apr 2013 20:06:18 +0000 Tue, 19 Feb 2013 00:03:59 +0000 Lustre 2.1.3 Lustre 2.4.0 0 6 <p>Recent change to MDT device size to only be 1G makes mkfs.lustre to not change default journal size to 400M (as that would be too big). Thus every test depending on larger journal would fail.</p> <p>We used to have a lower limit on the journal size in mkfs.lustre or mke2fs as well, because the minimum journal size was something very small like 4MB. It would be fine to make this test pass for >= 32MB as well, since the default journal size in mke2fs has increased to 32MB as well.</p> <p>That said, there seems to be something wrong with the test framework as well. The test checks that the MDSSIZE is large enough to run the test:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_18() { local MIN=2000000 local OK= # check <span class="code-keyword">if</span> current MDSSIZE is large enough [ $MDSSIZE -ge $MIN ] && OK=1 && myMDSSIZE=$MDSSIZE && \ log <span class="code-quote">"use MDSSIZE=$MDSSIZE"</span> : : [ -z <span class="code-quote">"$OK"</span> ] && skip_env <span class="code-quote">"$MDSDEV too small <span class="code-keyword">for</span> ${MIN}kB MDS"</span> && <span class="code-keyword">return</span> echo <span class="code-quote">"mount mds with large journal..."</span> local OLD_MDSSIZE=$MDSSIZE MDSSIZE=$myMDSSIZE reformat_and_config echo <span class="code-quote">"mount lustre system..."</span> : : } </pre> </div></div> <p>and the test output seems to confirm that the size should be large enough, but the actual format command doesn't seem to take the larger MDSSIZE into account (note <tt>262144</tt> at the end is the 1GB filesystem block count with 4kB blocks):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>use MDSSIZE=1073741824 mount mds with large journal... mke2fs -j -b 4096 -L lustre-MDT0000 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,huge_file,flex_bg -E lazy_itable_init,lazy_journal_init -F /dev/lvm-MDS/P1 262144 Writing CONFIGS/mountdata </pre> </div></div> <p>Looking into the reformat_and_config() function, it appears it doesn't use MDSSIZE at all. I suspect it looks like a bug was introduced with the landing of commit c04dddd71d140c61f654681c1993f5bb1ded4f48 (change <a href="http://review.whamcloud.com/2907" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2907</a>) that means this value is ignored. It used to be used for the "--device-size" argument to mkfs.lustre, and while code appears to be doing this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">var</span>=${type}SIZE <span class="code-keyword">if</span> [ -n <span class="code-quote">"${!<span class="code-keyword">var</span>}"</span> ]; then opts+=<span class="code-quote">" --device-size=${!<span class="code-keyword">var</span>}"</span> fi </pre> </div></div> <p>I guess this isn't working for some reason.</p> <p>As a work around I've changed the mds size to 2GB.</p> <p>I don't think that we could go much lower than 2GB anyway. In <a href="https://maloo.whamcloud.com/test_sessions/b55d305a-e4b8-11e1-9681-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/b55d305a-e4b8-11e1-9681-52540035b04c</a>, the "full" run was running out of space on the MDT in all of the parallel-scale tests.</p> <p>Bobijam</p> <p>I think that we are going to be able to live with the workaround for the 2.3 release itself but could you please investigate this in more detail so that we can more fully understand why this was failing and whether there is an issue that would be important in production environments</p> <p>Thanks</p> <p>Peter</p> <p>found the culprit </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> local MIN=2000000 ==============> we require at least 2G MDT # check if current MDSSIZE is large enough [ $MDSSIZE -ge $MIN ] && OK=1 && myMDSSIZE=$MDSSIZE && \ log "use MDSSIZE=$MDSSIZE" use MDSSIZE=1073741824 ===============> MDSSIZE has been defined to use 1T # check if the block device is large enough [ -z "$OK" ] && $(is_blkdev $SINGLEMDS $MDSDEV $MIN) && OK=1 && myMDSSIZE=$MIN && log "use device $MDSDEV with MIN=$MIN" we skipped this test, actually MDS device only has 1G (4k * 262144) space, and mkfs has no choice but format this 1G sized MDT and failed the test. </pre> </div></div> <p>we need always check block device's size bigger than $MIN.</p> <p>patch tracking at <a href="http://review.whamcloud.com/3632" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3632</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>patch description</b></div><div class="preformattedContent panelContent"> <pre>test: fix conf_sanity test_18 test case * We need always check block device's size bigger than its minimum requirement. * Correct code indentation for formatall(). </pre> </div></div> <p>Thanks for identifying this issue Bobijam. Hopefully the patch will still land in time for the 2.3 release but we do not need to track this as a blocker because the autotest workaround is preventing this causing test failures</p> <p>I'm not sure if the root cause has been identified here. In the very first Maloo report, the MDSSIZE is clearly wrong:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== conf-sanity test 18: check mkfs creates large journals == 02:01:09 (1344762069) use MDSSIZE=1073741824 </pre> </div></div> <p>Since the unit is KB, the value is around 1 TB. I think the test did the right thing, although with an incorrect MDSSIZE. Test Framework also passed the device size as expected:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: client-30vm3 /usr/sbin/mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1073741824 --mkfsoptions=\"-E lazy_itable_init\" --mountfsoptions=errors=remount-ro,iopen_nopriv,user_xattr,acl --reformat /dev/lvm-MDS/P1 </pre> </div></div> <p>See the "--device-size=1073741824" option? mkfs.lustre did not specify a journal size, which is what the code was written to do, it seems. And, mke2fs, as I have tried, chooses 32 MB in the case.</p> <p>If MDSSIZE were 1000000, the test should have called skip(). Hence, I don't think the test has to be changed.</p> <p>the problem here is what we specified MDT device size is bigger than what the MDT device actually has, and the MDT device real size does not meed the test's minimum requiremtn. The patch will detect this situation, even though we specified a big enough MDT size, we still check whethere the MDT device size is big enough.</p> <p>Ah, the problem is actually TT-820. I think the test does not need any fix; if all tests were to defend against test environment inconsistencies like this (i.e., MDSSIZE > block device size), we'd have a lot of work to do, right? The consistency checks should probably be done in Test Framework instead. For the moment, fixing TT-820 shall fix this test failure, no?</p> <p>I have the different idea, this specific test need a big enough MDT, so the specific test case need to make sure of that.</p> Related LU-1743 Development Rank 1|hzus4f: Rank (Obsolete) 2218 Severity