Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-10496] sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395 https://jira.whamcloud.com/browse/LU-10496 Lustre <p>sanity-dom sanity test 39k failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 39k: write, utime, close, stat ======================================================== 21:17:06 (1515705426) multiop /mnt/lustre/f39k.sanity voO_RDWR:w2097152_c TMPPIPE=/tmp/multiop_open_wait_pipe.15091 sanity test_39k: @@@@@@ FAIL: mtime is lost on close: 1515705427, should be 1484169395 </pre> </div></div> <p>Maloo reports:<br/> <a href="https://testing.hpdd.intel.com/test_sets/90c90676-f716-11e7-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/90c90676-f716-11e7-a6ad-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e1a7ef10-f6e6-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e1a7ef10-f6e6-11e7-bd00-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/fc89fcf6-f655-11e7-94c7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fc89fcf6-f655-11e7-94c7-52540065bddc</a></p> LU-10496 sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395 Bug Major Resolved Fixed Mikhail Pershin Jian Yu Fri, 12 Jan 2018 00:29:26 +0000 Fri, 26 Jan 2024 17:16:39 +0000 Wed, 9 Oct 2019 22:47:45 +0000 Lustre 2.11.0 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.13.0 Lustre 2.12.1 0 11 <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d7fa8c90-2d4e-11e8-b3c6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d7fa8c90-2d4e-11e8-b3c6-52540065bddc</a></p> <p>I'm able to reproduce this locally with ZFS (no DNE/DOM needed)</p> <p> </p> <p>another on master:</p> <p><a href="https://testing.whamcloud.com/test_sets/9665c524-920c-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9665c524-920c-11e8-b0aa-52540065bddc</a></p> <p>test does the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Process #1: multiop oO_RDWR:w2097152_c Process #2: wait #1 pause; setattr(mtime) </pre> </div></div> <p>It works as expected when <tt>setattr</tt> is being issued after <tt>write</tt> and before <tt>close</tt> as expected but sometimes it happens that <tt>write</tt> is not flushed yet and this bug occurs . It can be reproduced with changing <tt>multiop</tt> parameters as <tt>oO_RDWR:_w2097152c</tt></p> <p>I'd propose to make sure that write is finished by doing data sync with 'Y' parameter</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34029" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34029</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> test: update sanity tests 39 for DOM testing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a72572846e3461555affe0819c66b20a575d2459</p> <p>I've abandoned patch because it is incorrect replacement. Forced sync removes <tt>write</tt> vs <tt>setattr</tt> race and hides possible bug.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34115" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34115</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tests: disable 39k for DoM for a while<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f11c35de40d8d5abe1fed303268d61a73b739f0b</p> <p>Based on comments on Alex's last patch, the DoM code is not handling the file modification data (FMD) as is done in ofd. That tracks recent RPCs on each object and drops the modification time from writes that were generated before (lower XID) than setattr. </p> <p>AFAIU, Mike has been integrating FMD code into the target thing.</p> <p>yes, I working exactly on that</p> <p><del>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch:</del> <del><a href="https://review.whamcloud.com/34141" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34141</a></del><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tgt: move FMD functionality in target</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 3b6b66bc69d934dd794bf6e8529de9477ab982ce</del></p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34176" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34176</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> ofd: move FMD to the target code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3ec1fcde2ef53e59b03927d0c622be706f7d14cd</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34190" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34190</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tgt: move FMD handling from OFD to target<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 70e0e27f10b3e694997dc26095360fde1d7e31f3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34115/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34115/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tests: disable 39k for DoM for a while<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 68dd8a8acff9ad2295a1fcba318fc8ed5f140026</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34364" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34364</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tests: stop running sanity-dom/sanity 39k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8c797e5cb14145ca80013ec9266d7536b01b46df</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34176/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34176/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> ofd: move FMD to the target code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6459fa2b458612b5213b3b70839e340efff7aebc</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34190/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34190/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tgt: move FMD handling from OFD to target<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 52e33c507b84bcaf3af9df010f5de4a282aa3fca</p> <p>Landed for 2.13</p> <p><a href="https://testing.whamcloud.com/test_sets/5f72d680-5546-11e9-b98a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5f72d680-5546-11e9-b98a-52540065bddc</a> - but probably this is due to new locking. please ignore for a while.</p> <p>on b2_12 branch<br/> <a href="https://testing.whamcloud.com/test_sets/53e192d8-5b23-11e9-a256-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/53e192d8-5b23-11e9-a256-52540065bddc</a></p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34690" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34690</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> ofd: move FMD to the target code<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b21aafd60501902dadb6a48e876398b2db671380</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34691" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34691</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tgt: move FMD handling from OFD to target<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c98cd9b0bf0596c7943656008bb2567922ca864c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34690/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> ofd: move FMD to the target code<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 2eb6dee24a9f5804143da6258190387a3c50dee6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34691/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34691/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tgt: move FMD handling from OFD to target<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 5456b1bdbf94bdb1769000d4ca5a8131528ddf5d</p> <p>We are still seeing sanity-dom/sanity test 39k fail with the "mtime is lost on close" error message. A couple of recent failures on master are at<br/> <a href="https://testing.whamcloud.com/test_sets/73af0226-d2b5-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/73af0226-d2b5-11e9-9fc9-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/1ab44488-d2ce-11e9-a25b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1ab44488-d2ce-11e9-a25b-52540065bddc</a></p> <p>We were seeing an uptick in sanity-dom/sanity test 39k since moving to ZFS 0.8.1. We are going to revert to ZFS 0.7.13 and this may reduce or stop this test from failing. </p> <p>We will reopen this ticket if we see this test failing again once we are running ZFS 0.7.13.</p> <p> Another one on master:</p> <p><a href="https://testing.whamcloud.com/test_sets/c8b9f6de-d609-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c8b9f6de-d609-11e9-a2b6-52540065bddc</a></p> <p>Reopening ticket because we are seeing this issue with ZFS 0.7.13-1. Here are a couple of recent failures:<br/> <a href="https://testing.whamcloud.com/test_sets/26f6de30-d6b2-11e9-a25b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/26f6de30-d6b2-11e9-a25b-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/c8b9f6de-d609-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c8b9f6de-d609-11e9-a2b6-52540065bddc</a></p> <p>James, I wonder about the following - in test log I see that:<br/> <tt>excepting tests: 39k 42a 42b 42c 407 312 180</tt></p> <p>but test 39k is not in the test ALWAYS_EXCEPT list, so that looks like it is set by SANITY_EXCEPT, where is this set? And why 39k is excluded? I suppose that sanity-dom.sh ask to run 39k explicitly so it is ran but maybe it shouldn't?</p> <p>I agree that there is something not right with the SANITY_EXCEPT list in sanity-dom.sh. I will look <br/> into this issue. </p> <p>From your comment, it sound like sanity-dom/sanity test 39k should be skipped and this ticket should remain open until a fix can be found. If this is wrong, please let me know. </p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36352" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36352</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tests: enable 39k for DoM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 21d4843dec95997008416f26616ea0abb9b26054</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36228/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36228/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> ofd: serialize fmd check and set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 97a10cf9797bbed02fb131f6a205b6a0ceeb0525</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36352/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36352/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10496" title="sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395" class="issue-link" data-issue-key="LU-10496"><del>LU-10496</del></a> tests: enable 39k for DoM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf9bb79ebc5998c35a208ed011b64f9d4a62e7f3</p> <p>Looks like everything has landed for 2.13</p> Related LU-3285 LU-12710 LU-10589 LU-12967 Development Rank 1|hzzqzb: Rank (Obsolete) 9223372036854775807 Severity