https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11643 Lustre <p>We need to create disk images for Lustre 2.10.0 and 2.12.0 for conf-sanity test_32 upgrade regression testing, similar to the existing <tt>lustre/tests/disk*.tar.bz2</tt> files.</p> <p>These new test filesystems should include files using the new features for those releases, and tests that verify they are working properly:</p> <ul class="alternate" type="square"> <li>PFL file layouts</li> <li>project quotas</li> <li>FLR mirrored files</li> <li>DoM files on the MDTs</li> </ul> <p>Also, some of the files need to be striped over multiple OSTs (e.g. "<tt>lfs setstripe -c 2</tt>"), but store data on only a single object (i.e. size = 4KB). This relates to an LFSCK issue that I saw with <tt>filter_fid</tt> and would be good to test.</p> LU-11643

create disk images for Lustre 2.10 and 2.12 for ldiskfs

Question/Request Minor Resolved Fixed Sarah Liu Andreas Dilger LTS12 Thu, 8 Nov 2018 03:18:51 +0000 Thu, 26 Oct 2023 01:29:01 +0000 Mon, 7 Mar 2022 18:51:20 +0000 Lustre 2.12.0 Lustre 2.15.0 0 5 <p>Wei Liu (sarah@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35049" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35049</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images for upgrade tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b053a3ca368455db8c253c16b957d4271977d669</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37680" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37680</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> libext2fs: don't use O_DIRECT for files on tmpfs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: ad26a8f7d33f4d88a66a32b6898a71b518d9c0eb</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37680/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37680/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> libext2fs: don't use O_DIRECT for files on tmpfs<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: f94e4eef3f9d6a749798d4daddb6f67b708ea8c1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35049/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35049/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6b979daaffc36aeef145316b41d0e2fe8abcf20f</p> <p>Landed for 2.14</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38619" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38619</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: revert new images and tests for upgrade patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 07dc46059ec40cfb1b18e7f40bc35c185fc5ea33</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38619/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38619/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: revert new images and tests for upgrade patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ebaf3b1b9980bcea6e404e03dfd2ac48f21cc3fc</p> <p>Patch was reverted because of test failures.</p> <p>I have tried with disk2_12-ldiskfs image(2 mdts, 2 osts) again and found it hung in test_32a when try to mount the client without writeconf, client side shows following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[207084.287987] Lustre: DEBUG MARKER: == conf-sanity testing /usr/lib64/lustre/tests/disk2_12-ldiskfs.tar.bz2 upgrade ====================== 03:36:58 (1611286618) [207175.053537] Lustre: DEBUG MARKER: == conf-sanity nid = 10.9.6.32@tcp mount -t lustre 10.9.6.32@tcp:/t32fs /tmp/t32/mnt/lustre ========== 03:38:29 (1611286709) [207232.567915] LustreError: 30549:0:(mgc_request.c:1578:mgc_apply_recover_logs()) mgc: cannot find UUID by nid '10.9.6.32@tcp': rc = -2 [207232.570266] Lustre: 30549:0:(mgc_request.c:1797:mgc_process_recover_nodemap_log()) MGC10.9.6.32@tcp: error processing recovery log t32fs-cliir: rc = -2 [207232.572846] Lustre: 30549:0:(mgc_request.c:2149:mgc_process_log()) MGC10.9.6.32@tcp: IR log t32fs-cliir failed, not fatal: rc = -2 [207256.403162] LustreError: 30549:0:(lmv_obd.c:1263:lmv_statfs()) t32fs-MDT0000-mdc-ffff91bf366cf800: can't stat MDS #0: rc = -11 [207256.424154] Lustre: Unmounted t32fs-client [207256.425232] LustreError: 30549:0:(obd_mount.c:1680:lustre_fill_super()) Unable to mount (-11) [207312.563317] LustreError: 30588:0:(mgc_request.c:1578:mgc_apply_recover_logs()) mgc: cannot find UUID by nid '10.9.6.32@tcp': rc = -2 [207312.565523] LustreError: 30588:0:(mgc_request.c:1578:mgc_apply_recover_logs()) Skipped 1 previous similar message [207312.567342] Lustre: 30588:0:(mgc_request.c:1797:mgc_process_recover_nodemap_log()) MGC10.9.6.32@tcp: error processing recovery log t32fs-cliir: rc = -2 [207312.569687] Lustre: 30588:0:(mgc_request.c:1797:mgc_process_recover_nodemap_log()) Skipped 1 previous similar message [207312.571612] Lustre: 30588:0:(mgc_request.c:2149:mgc_process_log()) MGC10.9.6.32@tcp: IR log t32fs-cliir failed, not fatal: rc = -2 [207312.573790] Lustre: 30588:0:(mgc_request.c:2149:mgc_process_log()) Skipped 1 previous similar message [207322.606194] LustreError: 11-0: t32fs-MDT0000-mdc-ffff91bf793af000: operation mds_connect to node 10.9.6.32@tcp failed: rc = -11 [207402.734185] LustreError: 11-0: t32fs-MDT0000-mdc-ffff91bf793a8800: operation mds_connect to node 10.9.6.32@tcp failed: rc = -11 [207456.137501] LustreError: 30588:0:(lmv_obd.c:1263:lmv_statfs()) t32fs-MDT0000-mdc-ffff91bf793a8800: can't stat MDS #0: rc = -11 [207456.158467] Lustre: Unmounted t32fs-client [207456.159584] LustreError: 30588:0:(obd_mount.c:1680:lustre_fill_super()) Unable to mount (-11) </pre> </div></div> <p>The client mount code was added by <a href="https://jira.whamcloud.com/browse/LU-12846" title="could not delete orphan [0x2001ac5ff:0x3a30:0x0]: rc = -2" class="issue-link" data-issue-key="LU-12846"><del>LU-12846</del></a>, patch: <a href="https://review.whamcloud.com/#/c/37636/7" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/37636/7</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$MOUNT_CMD $nid:/$fsname $tmp/mnt/lustre || { error_noexit "Mounting the client" return 1 } [[ $(do_facet mds1 pgrep orph_.*-MDD | wc -l) == 0 ]] || error "MDD orphan cleanup thread not quit" umount $tmp/mnt/lustre || { error_noexit "Unmounting the client" return 1 } </pre> </div></div> <p>"Wei Liu &lt;sarah@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45827" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45827</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 413c3b75ba5fdfdc7e741489c3b0ba23c46e31c4</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45827/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45827/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f2143c0790bb1cb802fad7e81bcb386dc0245b36</p> <p>Landed for 2.15</p> <p>It looks like the newly landed images for 2.10 and 2.12 fail conf-sanity test_32a whenever they are run. The patch passed testing because it used:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: fstype=ldiskfs envdefinitions=ONLY="32f 32g" testlist=conf-sanit </pre> </div></div> <p>which did not run test 32a-e on the new images. The test is consistently hanging at:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>: CMD: onyx-41vm7 /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 CMD: onyx-41vm7 /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 CMD: onyx-41vm7 /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M CMD: onyx-41vm7 /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 CMD: onyx-41vm7 /usr/sbin/lctl pool_new t32fs.interop onyx-41vm7: Pool t32fs.interop created </pre> </div></div> <p>The next line of output for a passing test is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-65vm11 pgrep orph_.*-MDD </pre> </div></div> <p>There are still a number of patches that are currently passing this test because they have not yet updated to include this patch or are themselves marked <tt>trivial</tt> and do not run this test, and conversely conf-sanity was failing 100% on <tt>master-next</tt>.</p> <p>I will push a patch to temporarily disable the new images from being used for this test.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46353" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46353</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: skip some conf-sanity test_32 tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fcac417a66e49f99522e4d124783e43bb36f793b</p> <p>Is this fixed by the landing of <a href="https://review.whamcloud.com/46354/?" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46354/?</a></p> <p>test_32a hung with new disk images are known issue, I commented on <a href="https://jira.whamcloud.com/browse/LU-11643?focusedCommentId=290110&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-290110" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11643?focusedCommentId=290110&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-290110</a>, That "mount client" code was added by <a href="https://jira.whamcloud.com/browse/LU-12846" title="could not delete orphan [0x2001ac5ff:0x3a30:0x0]: rc = -2" class="issue-link" data-issue-key="LU-12846"><del>LU-12846</del></a>. And if I do writeconf, the mount can pass.</p> <p>Yes, the patch <a href="https://review.whamcloud.com/46354" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46354</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: replace nid in conf-sanity test_32</tt>" fixed the conf-sanity test_33a failure and allowed the mdt2 image to be mounted.</p> <p>The current problem is test_33b looks like the filesystem image is missing files in the <b>ROOT</b> directory that the test expects to see when there are multiple MDTs (<a href="https://jira.whamcloud.com/browse/LU-15506" title="conf-sanity test_32b failed with 2: sha1sum verification failed " class="issue-link" data-issue-key="LU-15506"><del>LU-15506</del></a>). It looks like the test is failing <b>before</b> it checks the <tt>striped_dir</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== checking sha1sums == CMD: onyx-71vm5 cat /tmp/t32/sha1sums /tmp/t32/mnt/lustre --- /tmp/t32/sha1sums.orig 2022-02-01 23:27:02.120240417 +0000 +++ /tmp/t32/sha1sums 2022-02-01 23:27:02.123240424 +0000 @@ -1,10 +0,0 @@ -59ced6686342e5fdff70a29277632622ad271168 ./init.d/functions -ff4f8d1bcd9ab4a9edcf77496e23963e5c6f6a2c ./init.d/lsvcgss -f8f634b92b75af4112634a6f14464e562cd82454 ./init.d/lustre -dff7d87de75271f0714c3b82921d40c96598f67a ./init.d/netconsole -21414c2b3c89f95d3eab00dafc954d3f6cf3ba9f ./init.d/network -f87a11aceaf7dc0e1614ea074fda14d6896ac66f ./init.d/README -92624163580750ca250a2c1cc8bd531d0609702a ./init.d/rhnsd -a17ecaeb91c0218092c8b01308a132698da9b81f ./pfl_dir/pfl_file -da39a3ee5e6b4b0d3255bfef95601890afd80709 ./project_quota_dir/pj_quota_file_old -2c72448b440f16c9fae18e287ca827c25d29a7cb ./rc.local ==** find returned files **== </pre> </div></div> <p>My patch <a href="https://review.whamcloud.com/46404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46404</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15506" title="conf-sanity test_32b failed with 2: sha1sum verification failed " class="issue-link" data-issue-key="LU-15506"><del>LU-15506</del></a> tests: improve conf-sanity test_32 messages</tt>" is adding some debugging to make it more clear which phases the test is running, and where the files are missing. I haven't had time to actually mount the disk2_10 filesystem locally and check whether the files are there, just doing everything via autotest/Maloo.</p> <p>The best solution would be fixing the <tt>test32newtarball()</tt> function to properly fill the image for what the tests expect, if that is really the problem, rather than fixing the images by hand.</p> <p>It also looks like the <tt>disk2_5-ldiskfs.tar.bz2</tt> image has a similar problem, which is causing the Janitor test failures, but it has never been tested because it wasn't included in the <tt>lustre-tests</tt> RPM due to not being added to <tt>lustre/tests/Makefile.am</tt>. I'm not sure whether it is worthwhile to fix that image at this point, or maybe it should be removed from Git entirely (though we would "lose" some test coverage in this case).</p> <p>Looking at the Janitor testing, it <b>also</b> failed test_32c in the same way, which is checking the <tt>striped_dir</tt>, but the files are missing, so it <em>may</em> not be an image problem. What is needed at this point is to check with master whether the mounted filesystem shows the files to be present in the root directory and <tt>striped_dir</tt>, and then check whether this matches 2.12, or if there really <b>is</b> an upgrade problem.</p> <p>For these 2 images, when doing the sha1sums check, it needs to go into the "remote_dir" dir instead of ROOT or striped_dir. I think we need to set the "pfl_upgrade=yes" (or other symbols) in test_32x which failed this part.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>if $r test -f $tmp/sha1sums; then # LU-2393 - do both sorts on same node to ensure locale # is identical $r cat $tmp/sha1sums | sort -k 2 &gt;$tmp/sha1sums.orig if [ "$dne_upgrade" != "no" ]; then pushd $tmp/mnt/lustre/striped_dir elif [ "$pfl_upgrade" != "no" ] || [ "$flr_upgrade" != "no" ] || [ "$dom_new_upgrade" != "no" ] || [ "$project_quota_upgrade" != "no" ]; then pushd $tmp/mnt/lustre/remote_dir else pushd $tmp/mnt/lustre fi </pre> </div></div> <p>I will try locally to verity first</p> <p>It looks like the test should be checking <b>both</b> <tt>remote_dir</tt> and <tt>striped_dir_old</tt>. It looks like a few parts of the test check for <tt>dne_upgrade</tt>, but they should always run that part of the test if <tt>mds2_is_available</tt> is true and the directory exists. It may be that we don't need to regenerate the images at all.</p> <p>Please make any changes starting with my patch <a href="https://review.whamcloud.com/46404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46404</a> so that we keep the debug messages. </p> <p>Fixed in 2.15</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sarah" class="user-hover" rel="sarah">sarah</a> , <br/> which code was used to create the disk2_12 image ? I am trying to add one more upgrade test (<a href="https://jira.whamcloud.com/browse/LU-16082" title="old-style Lustre EA inodes support is broken in newer kernel" class="issue-link" data-issue-key="LU-16082"><del>LU-16082</del></a>)  but cannot recreate the image easily.</p> <p>Hello,<br/> The test_32newtarball I used for 2.12 should be in the master code. </p> Duplicate LU-10998 Related LU-13514 LU-9207 LU-10215 LU-15751 LU-16082 LU-11625 LU-13514 LU-15506 LU-14772 LU-13604 LU-14853 LU-15482 Development Rank 1|i0060n: Rank (Obsolete) 9223372036854775807