https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12470 Lustre <p>This issue was created by maloo for Chris Horn &lt;hornc@cray.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/a0a32158-9756-11e9-8262-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a0a32158-9756-11e9-8262-52540065bddc</a></p> <p>test_47b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>create isn't blocked </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 47b: pdirops: remote mkdir vs create ================================================= 10:59:51 (1561460391) CMD: trevis-38vm9 lctl set_param fail_loc=0x80000145 fail_loc=0x80000145 CMD: trevis-38vm9 lctl set_param fail_loc=0 fail_loc=0 No conflict lfs mkdir: dirstripe error on '/mnt/lustre/f47b.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre/f47b.sanityn': File exists sanityn test_47b: @@@@@@ FAIL: create isn't blocked </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanityn test_41a - mkdir isn't blocked<br/> sanityn test_41b - create isn't blocked<br/> sanityn test_41c - link isn't blocked<br/> sanityn test_41d - unlink isn't blocked<br/> sanityn test_41e - rename isn't blocked<br/> sanityn test_41f - rename isn't blocked<br/> sanityn test_41g - getattr isn't blocked<br/> sanityn test_42a - mkdir isn't blocked<br/> sanityn test_42b - create isn't blocked<br/> sanityn test_42c - link isn't blocked<br/> sanityn test_42d - unlink isn't blocked<br/> sanityn test_42e - rename isn't blocked<br/> sanityn test_42f - rename isn't blocked<br/> sanityn test_42g - getattr isn't blocked<br/> sanityn test_43a - mkdir isn't blocked<br/> sanityn test_43b - create isn't blocked<br/> sanityn test_43c - link isn't blocked<br/> sanityn test_43d - unlink isn't blocked<br/> sanityn test_43e - rename isn't blocked<br/> sanityn test_43f - rename isn't blocked<br/> sanityn test_43g - getattr isn't blocked<br/> sanityn test_43i - remote mkdir isn't blocked<br/> sanityn test_44a - mkdir isn't blocked<br/> sanityn test_44b - create isn't blocked<br/> sanityn test_44c - link isn't blocked<br/> sanityn test_44d - unlink isn't blocked<br/> sanityn test_44e - rename isn't blocked<br/> sanityn test_44f - rename isn't blocked<br/> sanityn test_44g - getattr isn't blocked<br/> sanityn test_44i - remote mkdir isn't blocked<br/> sanityn test_45a - mkdir isn't blocked<br/> sanityn test_45b - create isn't blocked<br/> sanityn test_45c - link isn't blocked<br/> sanityn test_45d - unlink isn't blocked<br/> sanityn test_45e - rename isn't blocked<br/> sanityn test_45f - rename isn't blocked<br/> sanityn test_45g - getattr isn't blocked<br/> sanityn test_46a - mkdir isn't blocked<br/> sanityn test_46b - create isn't blocked<br/> sanityn test_46c - link isn't blocked<br/> sanityn test_46d - unlink isn't blocked<br/> sanityn test_46e - rename isn't blocked<br/> sanityn test_46f - rename isn't blocked<br/> sanityn test_46g - getattr isn't blocked<br/> sanityn test_46i - remote mkdir isn't blocked<br/> sanityn test_47a - mkdir isn't blocked<br/> sanityn test_47b - create isn't blocked<br/> sanityn test_47c - link isn't blocked<br/> sanityn test_47d - unlink isn't blocked<br/> sanityn test_47e - rename isn't blocked<br/> sanityn test_47f - rename isn't blocked<br/> sanityn test_47g - getattr isn't blocked</p> LU-12470

sanityn test_47b: create isn't blocked

Bug Major Resolved Fixed Andreas Dilger Maloo Tue, 25 Jun 2019 15:00:06 +0000 Thu, 23 Jun 2022 21:54:39 +0000 Thu, 23 Jan 2020 13:20:59 +0000 Lustre 2.13.0 Lustre 2.14.0 Lustre 2.14.0 0 10 <p>Ah, I get it.</p> <p>This the old "delay because ZFS sync is slow problem", where we're cancelling a pre-existing lock.  But we're clearing the client mdc locks first, so...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00010000:1.0:1561460391.953615:0:9926:0:(ldlm_lock.c:662:ldlm_add_bl_work_item()) ### lock incompatible; sending blocking AST. ns: mdt-lustre-MDT0000_UUID lock: ffff9a5196ab66c0/0x2d41583a090c8900 lrc: 2/0,0 mode: EX/EX res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x40000001000000 nid: 10.9.3.146@tcp remote: 0x7614c41d0ce57ab0 expref: 8 pid: 3617 timeout: 0 lvb_type: 0 </pre> </div></div> <p>This is the lock that MDT0 is cancelling...  And the owner: nid: 10.9.3.146@tcp </p> <p>Is another MDT.</p> <p>So we also have to clear MDT-MDT locks.  Easy enough.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35321" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35321</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: clear MDT-MDT locks for pdo tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4397884527e437439005606690489010b247d0dd</p> <p>Hopefully my change to the Maloo auto-vet works for this ticket. There were about 10 different tickets open for apparently identical failures on different subtests. I picked this one as the prime target because it has a potential fix.</p> <p>"There are about 10 different tickets open...", does that mean that this test fix could dramatically improve the pass rate of auto test? Is there any relation to the failures that prompted <a href="https://jira.whamcloud.com/browse/LU-12210" title="Test failures associated with DNE index and stripe count randomization" class="issue-link" data-issue-key="LU-12210">LU-12210</a> ?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=spitzcor" class="user-hover" rel="spitzcor">spitzcor</a>,</p> <p>No, these are all very low incidence, I think the highest is well below 1%, maybe below 0.1%.  But it's somewhere between 10 and 30 tests failing (I'd have to dig to be sure), so a lot of tickets get opened.</p> <p>Oh, and no, these tests are not among those that I noticed affected by <a href="https://jira.whamcloud.com/browse/LU-12210" title="Test failures associated with DNE index and stripe count randomization" class="issue-link" data-issue-key="LU-12210">LU-12210</a> when I looked in to it.  There's just a pile of tests that all do the same thing (testing blocking/not blocking behavior of different metadata ops, in DNE and non-DNE configs) and when Alex moved the wait time in these tests from "obscenely long" to "reasonable" many of them started getting a few different low probability failures, some common to the full set, some limited to DNE.</p> <p>I don't really know if the patch I've got here is the last problem, but I'm hopeful.  Anyway, even though there's a lot of different tests, the aggregate failure rate is not that high, and we're currently passing autotest pretty reliably.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35321/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35321/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: clear MDT-MDT locks for pdo tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 43ed7101e10e395839f9406bead6a5ac4fb02997</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35865" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35865</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: clear MDT-MDT locks for pdo tests<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 5c668c55ce8e44520f4eec7b1ae35eceeaa3a75b</p> <p>Looks like this issue is not totally resolved - <a href="https://testing.whamcloud.com/test_sets/5cafca60-c5a9-11e9-98c8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5cafca60-c5a9-11e9-98c8-52540065bddc</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pfarrell" class="user-hover" rel="pfarrell">pfarrell</a> re-open this ticket or open a new one?</p> <p>Opened <a href="https://jira.whamcloud.com/browse/LU-12689" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-12689</a> to track new failure</p> <p>It seems that there are now two different failure modes for these tests:</p> <ul class="alternate" type="square"> <li>occasional review test failures for patches similar to what happened before the <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> patch landed, so it isn't clear whether this patch really fixed anything?</li> <li>complete failure of subtests in the 40-46 range when running tests with 2.12.2 MDS/OSS and reporting locks <b>not</b> being canceled by <tt>lru_size=clear</tt>: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 41b: pdirops: create vs create ======================================================= 15:25:02 (1565882702) CMD: trevis-20vm12 /usr/sbin/lctl set_param -n ldlm.namespaces.*mdt*.lru_size=clear CMD: trevis-20vm12 /usr/sbin/lctl get_param ldlm.namespaces.*mdt*.lock_unused_count ldlm.namespaces.*mdt*.lock_count ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=43 </pre> </div></div></li> </ul> <p>The testing interop problem <em>may</em> relate to a patch in 2.13 on how the LRU is cleared (always waiting for locks to be cancelled?), but I can't find the patch in question.</p> <p>I think it makes sense to reopen this patch and <b>not</b> mark it resolved for 2.13, and either revert the patch (since it breaks testing interop), or figure out what is causing the testing interop issues and include the fix to 2.12.3 (though this is sub-optimal for some reasons).</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35865/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35865/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: clear MDT-MDT locks for pdo tests<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: cf3506ccb25f4a4f57150e51e7a8bde4cee80a26</p> <p>+1 on master with the patch: <a href="https://testing.whamcloud.com/test_sets/60b8ee10-e03a-11e9-a197-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/60b8ee10-e03a-11e9-a197-52540065bddc</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/3f181300-d3f8-465a-88de-95756bf58f3c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/3f181300-d3f8-465a-88de-95756bf58f3c</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/7a906b8b-f8ee-48b5-91bc-d3628a032560" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/7a906b8b-f8ee-48b5-91bc-d3628a032560</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/bca0aec6-0abe-11ea-b934-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/bca0aec6-0abe-11ea-b934-52540065bddc</a></p> <p>Seen on master: <a href="https://testing.whamcloud.com/sub_tests/863eeb94-2b1c-11ea-b0f4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/863eeb94-2b1c-11ea-b0f4-52540065bddc</a></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37304" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37304</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: increase pdirops timeout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a42f438a4acac3145cd5b7b178a737dbacc43d32</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37304/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37304/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12470" title="sanityn test_47b: create isn&#39;t blocked" class="issue-link" data-issue-key="LU-12470"><del>LU-12470</del></a> tests: increase pdirops timeout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b35f50c96c608ba650a5b3cf29fa129e01025549</p> <p>I'm going to open a new ticket for this, but thought I'd leave a note here. We are still seeing this issue for interop testing between master clients and 2.12.6 servers. </p> <p>One such failure is at <a href="https://testing.whamcloud.com/test_sets/d3a83e0b-15be-4799-b8ba-c264835472e2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d3a83e0b-15be-4799-b8ba-c264835472e2</a> .</p> <p>... and that bug is <a href="https://jira.whamcloud.com/browse/LU-14746" title="Interop: sanityn test_41b fails with &#39;create isn&#39;t blocked&#39;" class="issue-link" data-issue-key="LU-14746">LU-14746</a></p> Duplicate LU-12407 LU-10870 LU-14746 LU-4575 LU-6641 LU-8759 LU-10874 LU-12408 LU-12435 LU-12437 LU-12449 LU-12466 LU-12551 LU-12576 LU-13097 Related LU-12689 LU-10754 Development Rank 1|i00ipb: Rank (Obsolete) 9223372036854775807 Severity