https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7009 Lustre <p>I think the best approach here is to run the DNE-on-ZFS testing a few times to see which test scripts are passing regularly, and then move the passing scripts (or passing with minimal additions to ALWAYS_EXCEPT) to review-zfs-part-1 running with 4 MDTs and leave the failing tests (if not too many) in review-zfs-part-2 running with 1 MDT until we can fix the remaining test issues. As tests pass they can be moved into review-zfs-part-1 until it gets too large, and then just add the remaining few tests to ALWAYS_EXCEPT and set review-zfs-part-2 to also run with 4 MDTs and rebalance the tests to run with approximately the same time.</p> <p>That way we can start basic DNE-on-ZFS testing ASAP and migrate the remaining ZFS tests to DNE without having to skip many of the tests.</p> <p>Of course, if DNE-on-ZFS is mostly or completely passing today then this incremental migration to DNE testing can be skipped.</p> LU-7009

Testing DNE on ZFS

Bug Minor Resolved Done James Nunez Di Wang Fri, 14 Aug 2015 21:23:26 +0000 Fri, 15 Nov 2019 15:14:14 +0000 Fri, 15 Nov 2019 15:14:14 +0000 Lustre 2.8.0 0 11 <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16122" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16122</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7009" title="Testing DNE on ZFS" class="issue-link" data-issue-key="LU-7009"><del>LU-7009</del></a> test: test DNE on ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fe93134b19331e9300f339c4584390b37fced368</p> <p>See also <a href="https://testing.hpdd.intel.com/test_sessions/7f7566d8-d281-11e4-a20f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/7f7566d8-d281-11e4-a20f-5254006e85c2</a> which has results from <a href="http://review.whamcloud.com/14148" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14148</a> but back in March. Not sure if those test results are all still available, but at least it provides one more data point for this testing.</p> <p>Met this issue when testing on shadow</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:31:31:Loading modules from /usr/lib64/lustre 19:31:31:detected 2 online CPUs by sysfs 19:31:31:Force libcfs to create 2 CPU partitions 19:31:31:debug=vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck 19:31:31:subsystem_debug=all -lnet -lnd -pinger 19:31:33:Formatting mgs, mds, osts 19:31:33:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:35:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:35:Format mgs: lustre-mgs/mgs 19:31:35:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:36:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:36:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:37:CMD: shadow-50vm3 grep -c /mnt/mgs' ' /proc/mounts 19:31:37:CMD: shadow-50vm3 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' 19:31:37:/usr/lib64/lustre/tests/test-framework.sh: line 3157: [: too many arguments 19:31:38:CMD: shadow-50vm3 ! zpool list -H lustre-mgs &gt;/dev/null 2&gt;&amp;1 || 19:31:38: grep -q ^lustre-mgs/ /proc/mounts || 19:31:38: zpool export lustre-mgs 19:31:39:CMD: shadow-50vm3 mkfs.lustre --mgs --param=sys.timeout=20 --backfstype=zfs --device-size=2097152 --reformat lustre-mgs/mgs /dev/lvm-Role_MDS/P1 19:31:39:shadow-50vm3: 19:31:40:shadow-50vm3: mkfs.lustre FATAL: unhandled fs type 5 'zfs' 19:31:40:shadow-50vm3: 19:31:40:shadow-50vm3: mkfs.lustre FATAL: unable to prepare backend (22) 19:31:40:shadow-50vm3: mkfs.lustre: exiting with 22 (Invalid argument) 20:31:24:********** Timeout by autotest system ********** </pre> </div></div> <p>Looks like a TEI issue.</p> <p>This testing is complete and tickets were created for the issues found</p> <p>I'd rather this not be closed until we are actually testing DNE+ZFS automatically via autotest. </p> <p>Per the original comment, if there are tests that are already passing them they can be moved into an autotest config file separate from the tests that are currently failing and/or disabled. If enough tests are passing, we could just rearrange review-zfs-part-</p> {1,2} <p> and change the config for one of them to start testing with MDSCOUNT=2.</p> <p>It makes sense to start doing this as soon as possible to avoid regressions while the existing bugs are fixed or disabled. </p> <p>That's a good point Andreas. We'd need to land either fixes for the bugs or else temporarily disable the affected tests while the fix was being worked on and then make the TEI request. Should we start by introducing that change on full test runs first and then rolling out to master after a few successful runs?</p> <p>I'm reluctant to break "full" runs more than they already are today. Also, history shows that tests running on master get far less attention than those running on review. Rather, I think the existing testing patch can be expanded until it starts passing or skipping (for only the ZFS+DNE config) all the tests currently being run for review-zfs-part-</p> {1,2} <p> and then landed like any other patch.</p> <p>Since it would only be adding exceptions for the ZFS+DNE tests it couldn't cause breakage when it lands. It should be including full runs of those tests so should be OK to enable the DNE config afterward, as long as we leave a non-DNE test running for the excepted or skipped tests, as we do with review-ldiskfs today. </p> <p>We are running the 'full' test group with ZFS + DNE. Some recent results are at:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/9803525a-f453-11e6-b655-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/9803525a-f453-11e6-b655-5254006e85c2</a> <br/> <a href="https://testing.hpdd.intel.com/test_sessions/d38b0f30-0262-11e7-a4d2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d38b0f30-0262-11e7-a4d2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/d5adbff2-096e-11e7-b5b2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d5adbff2-096e-11e7-b5b2-5254006e85c2</a></p> <p>There are many failures, but there are a few that consistently fail:<br/> sanity-lfsck test_5e (<a href="https://jira.whamcloud.com/browse/LU-8840" title="sanity-lfsck test_2e: @@@@@@ FAIL: (5) Fail to repair crashed linkEA: 0" class="issue-link" data-issue-key="LU-8840"><del>LU-8840</del></a>)<br/> sanity-quota test_12b (Ticket ??)</p> <p>Rather than doing "full" test runs (which we know have permanent failures even with ldiskfs), it would be better to run <tt>review-dne-part-1</tt> and <tt>review-dne-part-2</tt> with <tt>mdtfilesystemtype=zfs</tt> and <tt>ostfilesystemtype=zfs</tt> (or <tt>review-zfs-part-1</tt> and <tt>review-zfs-part-2</tt> with <tt>mdscount=4</tt>), so that we can have an apples-to-apples comparison of tests that are failing only with ZFS.</p> <p>I see that there is also a regular <tt>replay-single</tt> test failure, but other than that and the two failures mentioned above, I think there are enough tests that ARE passing that we should create a new test session <tt>review-zfs</tt> that has only the failing DNE ZFS tests, and change the main ZFS test sessions to have multiple MDTs/MDS. It would be good to do this now that we are past the 2.10 feature freeze and before the release so that we have high confidence in this code and there are no (further?) regressions.</p> <p>James, not sure if this is already duplicate with another ticket you are using to track this work?</p> <p>James, I think this work is done now? We are always testing review-dne-zfs-partX for each patch. I think this can be closed.</p> <p>There are multiple test groups that run in a DNE environment namely review-dne-*<del>part</del></p> {1,2,3,4} <p>. Thus, this ticket can be closed.</p> Related LU-7191 LU-6831 LU-7192 Development Rank 1|hzxkkn: Rank (Obsolete) 9223372036854775807 Severity