https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14600 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/84000ede-1df1-4f8f-90a0-44f5afc1ea05" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/84000ede-1df1-4f8f-90a0-44f5afc1ea05</a></p> <p>test_30 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>stat: cannot stat '/mnt/lustre/d30.sanity-lfsck/foo/f0': No such file or directory (18) f0 is not recovered </pre> </div></div> <p>This only started failing on 2021-04-08, so is very likely a regression due to a recent landing. </p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lfsck test_30 - (18) f0 is not recovered</p> LU-14600

sanity-lfsck test_30: f0 is not recovered

Bug Minor Resolved Fixed Andreas Dilger Maloo Fri, 9 Apr 2021 15:28:31 +0000 Fri, 16 Apr 2021 23:40:16 +0000 Fri, 16 Apr 2021 07:02:28 +0000 0 7 <p>What is very strange is that this failure started happening on several different branches on the same day - master, b2_12, b_es5_2, and b_es6_0, which makes it unlikely (though not impossible) to be caused by the same patch landing on all three branches at once. </p> <p><del>Testing on 2021-04-09 looks like it has started passing again</del>, so it is <em>possible</em> it was some kind of date-related bug (unlikely, but it happened with <a href="https://jira.whamcloud.com/browse/LU-13314" title="sanity test_56ob: lfs find /mnt/lustre/d56ob.sanity -mtime 1y&#39; wrong: found 0, expected 1" class="issue-link" data-issue-key="LU-13314"><del>LU-13314</del></a>), or a strange hiccup in the test environment that only caused this particular test to fail that day?</p> <p>Seen on master. <a href="https://testing.whamcloud.com/test_sets/6fd26096-1e3c-4fbc-8685-d79e6f682975" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6fd26096-1e3c-4fbc-8685-d79e6f682975</a></p> <p>Seen on b2_12:  <a href="https://testing.whamcloud.com/test_sessions/0b38adbc-8f78-4b49-8163-19007be5e8c7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/0b38adbc-8f78-4b49-8163-19007be5e8c7</a><font color="#2e3436"> </font><br/>                             <a href="https://testing.whamcloud.com/test_sessions/eb8ea52e-1afe-4364-956f-1d424be88c97" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/eb8ea52e-1afe-4364-956f-1d424be88c97</a></p> <p>This patch only started failing on 2021-04-08, and this is now a 100% failure for <tt>review-dne-part-2</tt> and <tt>full</tt> sessions for both master and b2_12 (the <tt>review-dne-zfs-part-2</tt> sessions are passing because this test is ldiskfs-only).</p> <p>There were several patches landed to b2_12 on 2021-04-06:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>f735003c0f LU-14355 ptlrpc: do not output error when imp_sec is freed 0596a16841 LU-12506 changelog: support large number of MDT 7f04890a1b LU-13609 mgs: fix config_log buffer handling 0850c7b14a LU-13649 mdd: orhpan cleanup fix 5610ef9a7a LU-1538 tests: standardize test script init - sanity 7531c5d25c LU-14450 kernel: kernel update RHEL8.3 [4.18.0-240.15.1.el8_3] 2fd278af4c LU-11518 ldlm: lru code cleanup eaee7c3cd6 LU-11518 osc: cancel osc_lock list traversal once found the lock is being used </pre> </div></div> <p>Patches landed to master on 2021-04-06 are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>622e4c6e04 LU-14547 test: skip sanityn 109 for local setup 14a1102268 LU-14552 ptlrpc: NULL pointer dereference in ptlrpc_watchdog_fire f9d837b479 LU-14540 o2iblnd: Use REMOTE_DROPPED for ECONNREFUSED 3f8a6fd7d6 LU-14538 gss: make namespace optional in lgss_keyring 9cc7128b9b LU-14522 ldlm: reprocess locks if enqueue failed 1d3c585194 LU-14487 lustre: remove references to Sun Trademark. 642682a39e LU-14450 kernel: kernel update RHEL8.3 [4.18.0-240.15.1.el8_3] f37bce8a57 LU-14119 osd: add mount option "resetoi" 99d00b97ef LU-14119 osd: delete stale OI mapping entry f5136e8195 LU-14119 osd-zfs: enable LUDA_VERIFY bf47526261 LU-14119 mdc: set fid2path RPC interruptible 771308ada3 LU-14291 ptlrpc: format UPDATE messages in server-only code 67d17dd590 LU-14195 libcfs: switch to kfree_sensitive d7249d9d70 LU-13783 libcfs: provide fallback kallsyms_lookup_name() 3d101645a5 LU-14132 lod: do not initialize sub llogs twice 00141b1a74 LU-11776 utils: add support lfs find with mdt hash flag 4126fbb30c LU-13397 lfs: mirror resync to keep sparseness 77f5bb4dac LU-6142 lustre: convert IFTODT to S_DT f38f09e02a LU-14090 mgs: no local logs flag 2a34dc95bd LU-12142 clio: fix hang on urgent cached pages 1058867c00 LU-12142 readahead: limit over reservation b4391fcdaf LU-10632 tests: recovery-small test_26 idle_timeout </pre> </div></div> <p>so the only common patch between these two branches is the <tt><a href="https://jira.whamcloud.com/browse/LU-14450" title="kernel update [RHEL8.3 4.18.0-240.15.1.el8_3]" class="issue-link" data-issue-key="LU-14450"><del>LU-14450</del></a> kernel: kernel update RHEL8.3 <span class="error">&#91;4.18.0-240.15.1.el8_3&#93;</span></tt> patch (update from kernel 4.18.0-240.1.1.el8), but that is only affecting the <b>client</b> for b2_12 testing.</p> <p>Comparing the test environment for the last passing and first failing run on b2_12 shows the server kernel version is different:<br/> <span class="error">&#91;https://testing.whamcloud.com/test_sessions/b4203201-088d-47c8-87be-e08ef8e31cf1|2021-04-07 05:11:32&#93;</span>: pass Kernel version 3.10.0-1160.15.2.el7_lustre.x86_64<br/> <span class="error">&#91;https://testing.whamcloud.com/test_sessions/bb45aa98-1c36-492d-a0f6-4b61135fae40|2021-04-09 22:57:14&#93;</span>: fail Kernel version 3.10.0-1160.21.1.el7_lustre.x86_64</p> <p>However, on master, both the passing and failing runs are using the same RHEL8 kernel on the servers:<br/> <span class="error">&#91;https://testing.whamcloud.com/test_sessions/e72881e6-7c88-4fe3-888c-a431d2ad5810|2021-04-08 04:23:19&#93;</span>:pass Kernel version 4.18.0-240.15.1.el8_lustre.x86_64<br/> <span class="error">&#91;https://testing.whamcloud.com/test_sessions/94bf8f9f-b129-46aa-bdde-8405a3216f63|2021-04-08 11:00:47&#93;</span> Kernel version 4.18.0-240.15.1.el8_lustre.x86_64</p> <p>so it looks like this is caused somehow by a test environment change that happened on 2021-04-08 between 04:23-11:00.</p> <p>The only other common element among the test sessions is e2fsprogs, and it appears that the landing of patch <a href="https://review.whamcloud.com/43169" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43169</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11446" title="ldiskfs inodes nlink mismatch with DNE" class="issue-link" data-issue-key="LU-11446">LU-11446</a> e2fsck: check trusted.link when fixing nlink</tt>" has caused this test to start failing.</p> <p>The passing sessions are reporting e2fsprogs with "<tt>e2fsck 1.45.6.wc5 (09-Feb-2021)</tt>" and failing ones report "<tt>e2fsck 1.45.6.wc6 (09-Apr-2021)</tt>".</p> <p>Reading the test description for <tt>sanity-lfsck.sh::test_30()</tt> makes it clear that this is the cause, because the fix to <tt>e2fsck</tt> is specifically to <b>avoid</b> the entry being moved to <tt>lost+found</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Inject failure stub on MDT0 to simulate the case that directory d0 has no linkEA entry, then the LFSCK will move it into .lustre/lost+found/MDTxxxx/ later. : Pass 4: Checking reference counts -Unattached inode 183 -Connect to /lost+found? yes - -Inode 183 ref count is 2, should be 1. Fix? yes - Unattached inode 187 Connect to /lost+found? yes Inode 187 ref count is 2, should be 1. Fix? yes -Unattached inode 192 -Connect to /lost+found? yes - -Inode 192 ref count is 2, should be 1. Fix? yes - -Unattached inode 193 -Connect to /lost+found? yes - -Inode 193 ref count is 2, should be 1. Fix? yes - Unattached inode 199 Connect to /lost+found? yes Inode 199 ref count is 2, should be 1. Fix? yes Inode 20106 ref count is 1, should be 2. Fix? yes Inode 20108 ref count is 3, should be 2. Fix? yes </pre> </div></div> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/43324" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43324</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14600" title="sanity-lfsck test_30: f0 is not recovered" class="issue-link" data-issue-key="LU-14600"><del>LU-14600</del></a> e2fsck: check trusted.link after linking inode<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 2<br/> Commit: b83f196e93f3cb982f720aeced810cbf650cde04</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43324/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43324/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14600" title="sanity-lfsck test_30: f0 is not recovered" class="issue-link" data-issue-key="LU-14600"><del>LU-14600</del></a> e2fsck: check trusted.link after linking inode<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 87164b117be3bdeb1becf1960b80687637eda08f</p> <p>This patch appears to have fixed the problem - all four review-dne-part-2 sanity-lfsck runs started after 4am MT have passed. </p> <p>What is still needed here is an e2fsck test case for this - unreferenced inodes with xattrs that need to be relinked to lost+found. </p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/43335" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43335</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14600" title="sanity-lfsck test_30: f0 is not recovered" class="issue-link" data-issue-key="LU-14600"><del>LU-14600</del></a> e2fsck: trusted.link unref inode test case<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: e12d23d0241a1d09e05c1ef129f201c8c1515ffa</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43335/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43335/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14600" title="sanity-lfsck test_30: f0 is not recovered" class="issue-link" data-issue-key="LU-14600"><del>LU-14600</del></a> e2fsck: trusted.link unref inode test case<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: f46cb5c041147772639c56d993a4313e2655399d</p> <p>Fixed in 1.45.6.wc7</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/43352" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43352</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14600" title="sanity-lfsck test_30: f0 is not recovered" class="issue-link" data-issue-key="LU-14600"><del>LU-14600</del></a> misc: update to e2fsprogs-1.45.6.wc7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1538b8f6d8c31143e059dc95c3724d01d9f93a13</p> Related LU-11446 Development Rank 1|i01rsf: Rank (Obsolete) 9223372036854775807 Severity