https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10754 Lustre <p>sanityn test_47b fails for DNE/ZFS testing. From the client test_log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 47b: pdirops: remote mkdir vs create ================================================= 09:50:51 (1519725051) CMD: onyx-32vm9 lctl set_param fail_loc=0x80000145 fail_loc=0x80000145 sanityn test_47b: @@@@@@ FAIL: create must fail lfs mkdir: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/f47b.sanityn' (3): stripe already set lfs setdirstripe: cannot create stripe dir '/mnt/lustre/f47b.sanityn': File exists </pre> </div></div> <p>This test started failing on 2018-01-19.</p> <p>The console and dmesg logs don’t have any more information in them for this failure than what is in the suite_log.</p> <p>Logs for test sessions with this failure are at<br/> <a href="https://testing.hpdd.intel.com/test_sets/b9932fbe-fd24-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9932fbe-fd24-11e7-bd00-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/316d208c-1bb4-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/316d208c-1bb4-11e8-a7cd-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/5ce20dce-1905-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ce20dce-1905-11e8-a6ad-52540065bddc</a></p> LU-10754

sanityn test 47b fails with 'create must fail'

Bug Major Resolved Fixed Patrick Farrell James Nunez dne zfs Fri, 2 Mar 2018 00:22:09 +0000 Tue, 26 Oct 2021 18:49:48 +0000 Wed, 22 May 2019 21:54:32 +0000 Lustre 2.11.0 Lustre 2.12.2 Lustre 2.13.0 0 7 <p><a href="https://testing.whamcloud.com/test_sessions/b9d6b421-a589-4a3c-8daa-a9e2e08e5b31" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/b9d6b421-a589-4a3c-8daa-a9e2e08e5b31</a></p> <p><a href="https://testing.whamcloud.com/test_sessions/bc2f83f6-83cc-4baa-8854-25a3f29ecb0c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/bc2f83f6-83cc-4baa-8854-25a3f29ecb0c</a></p> <p>So what I learned after looking at these logs for a bit is that the DNE2 protocol is complex and I don't understand it, but what I did learn is that we are hanging such that the lfs mkdir is not running until after the multiop has started:</p> <p>&#8212;</p> <p>$LFS mkdir -i 1 $DIR1/$tfile &amp;<br/> PID1=$!<br/> sleep 1<br/> multiop $DIR2/$tfile oO_CREAT:O_EXCL:c &amp;&amp; error "create must fail"</p> <p>&#8212;</p> <p>The lfs mkdir hangs waiting on a lock (not the intended PDO lock, I think - much earlier...?).  The sequence of events is a bit beyond me right now, but it involves cancelling locks on MDT1, which are presumably held because of previous tests.  Somewhere in there, something gets stuck, and the lock cancellation does not complete (on MDT0) until after the create operation has started, hence the mkdir failing with EEXIST.</p> <p>I can't prove it, but I have a guess on this.</p> <p>The lock it's waiting for was created by/during a previous test, and I can't prove it, but I think there's some uncommitted stuff under it that's getting sync'ed.  In that case, 'sleep 1' simply isn't long enough for that to happen due to our lack of the ZIL and the long commit intervals on ZFS (once per second).</p> <p> </p> <p>I'm going to push a patch to change the sleeps to 2 seconds.</p> <p>Discussed with <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a>, he's going to take a look and see what we can do, maybe we can do better than just increasing the sleep.  (Which isn't 100% reliable, but we also want to avoid syncs...)</p> <p>What about just cancelling all of the MDC locks at the start of the test, so that we aren't waiting on that in the middle of the test? That avoids making the sleep to be long enough for the 1% case, without wasting time to make it too long for 99% of runs. We still need a small sleep due to fork/exec, but that can be <tt>sleep 0.2</tt> or similar.</p> <p>That makes a <b>lot</b> more sense than my suggestion.</p> <p>Andreas, has this failed recently?  I ignored it because I saw it a few times, then hadn't seen it since.</p> <p>PS: I post here because it seems like there has been an uptick in the frequency of test failures:</p> <ul class="alternate" type="square"> <li>2019-05-10: 4 failures already today</li> <li>2019-05-09: 4 failures the week ending on</li> <li>2019-05-02: 2 failures that week</li> <li>2019-04-25: 2 failures that week</li> </ul> <p>There were a number of patches landed on 2019-05-08, so it is likely one of them is involved (possibly just increasing the size of the race window by making some piece of code/test slower.</p> <p>Presumably:</p> <p><a href="https://review.whamcloud.com/4392" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/4392</a></p> <p>Which modifies these <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>It looks like the recent uptick in failures is caused by the landing on 2019-05-10 of patch <a href="https://review.whamcloud.com/4392" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/4392</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-2233" title="sanityn/40-46 takes ~800-900s" class="issue-link" data-issue-key="LU-2233"><del>LU-2233</del></a> tests: improve tests sanityn/40-47</tt>", which reduced the "sleep" time to 0.2s, which increased the probability of other lock<br/> operations still being blocked when this test was run.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34848" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34848</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10754" title="sanityn test 47b fails with &#39;create must fail&#39; " class="issue-link" data-issue-key="LU-10754"><del>LU-10754</del></a> tests: Clear mdc locks before tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1a4b819075e2936ed9dc5d4b3600a7be37270c8a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34848/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34848/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10754" title="sanityn test 47b fails with &#39;create must fail&#39; " class="issue-link" data-issue-key="LU-10754"><del>LU-10754</del></a> tests: Clear mdc locks before tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 26a7abe812834663a6d42ba0d04606650c71cb9f</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34853" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34853</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10754" title="sanityn test 47b fails with &#39;create must fail&#39; " class="issue-link" data-issue-key="LU-10754"><del>LU-10754</del></a> debug: try to sleep for 1s<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 24860d1c1948a2a7ccec0be8eb1ea8ee78e63cb6</p> <p>This is causing about 1/3 of each test suite runs to fail.</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34853/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34853/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10754" title="sanityn test 47b fails with &#39;create must fail&#39; " class="issue-link" data-issue-key="LU-10754"><del>LU-10754</del></a> tests: sanityn/47b to sleep for 1s<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 85ac0da36748311528aa7b89f62679f087b7445c</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35399" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35399</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10754" title="sanityn test 47b fails with &#39;create must fail&#39; " class="issue-link" data-issue-key="LU-10754"><del>LU-10754</del></a> tests: Debug full testing failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c6779689cbcd658b600933425db1d5365b3a3337</p> <p>OK, figured it out:<br/> <a href="https://review.whamcloud.com/#/c/4392/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/4392/</a><br/> breaks interop.</p> <p>Wondering if we can just land it to b2_12?</p> <blockquote> <p>Wondering if we can just land it to b2_12?</p></blockquote> <p>If this patch is landed to b2_12, will it then cause interop testing problems for 2.12 vs. 2.10? Not that I'm totally against this, since we run far more testing on master than b2_12, but ideally we should just add a version interop check, or just skip this test for interop testing on b2_12 so that there aren't gratuitous errors that need to be looked at.</p> <p>The failure still occurred on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/c376ce50-b86c-11e9-b753-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c376ce50-b86c-11e9-b753-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/3e5c96d8-b7c6-11e9-b88c-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3e5c96d8-b7c6-11e9-b88c-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/29aab538-b66a-11e9-b88c-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/29aab538-b66a-11e9-b88c-52540065bddc</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a>,</p> <p>There are a number of bugs for this, unfortunately.  Here's the one we're using now, which has an unlanded fix: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a></p> <p>Thank you, Patrick.</p> Duplicate LU-13097 Related LU-12470 LU-2233 Development Rank 1|hzztnj: Rank (Obsolete) 9223372036854775807 Severity