https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3180 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/ff811592-a66a-11e2-90ad-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/ff811592-a66a-11e2-90ad-52540035b04c</a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>03:53:10:Memory used: 2436k/21180k (745k/1692k), time: 0.20/ 0.07/ 0.02 03:53:10:I/O read: 10MB, write: 0MB, rate: 50.10MB/s 03:53:10:CMD: client-19vm1.lab.whamcloud.com PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh _check_progs_installed lfsck 03:53:10:CMD: client-19vm1.lab.whamcloud.com PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin: NAME=autotest_config sh rpc.sh is_mounted /mnt/lustre 03:53:10:lfsck -c -l --mdsdb /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/mdsdb --ostdb /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-0 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-1 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-2 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-3 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-4 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-5 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-6 /mnt/lustre 03:53:11:CMD: client-19vm1.lab.whamcloud.com lfsck -c -l --mdsdb /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/mdsdb --ostdb /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-0 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-1 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-2 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-3 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-4 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-5 /home/autotest/.autotest/shared_dir/2013-04-14/224508-70192991849440/ostdb-6 /mnt/lustre 03:53:11:lfsck 1.42.6.wc2 (10-Dec-2012) 03:53:11:lfsck: ost_idx 0: pass1: check for duplicate objects 03:53:11:lfsck: ost_idx 0: pass1 OK (12 files total) 03:53:11:lfsck: ost_idx 0: pass2: check for missing inode objects 03:53:11:Failed to find fid [0x2000013a1:0xda11:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda15:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda16:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda13:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda12:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda14:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda17:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda18:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda19:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda1b:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda1a:0x0]: DB_NOTFOUND: No matching key/data pair found 03:53:11:Failed to find fid [0x2000013a1:0xda1c:0x0]: DB_NOTFOUND: No matching key/data pair found </pre> </div></div> server and client: tag-2.3.64 build #1411 LU-3180

Test failure on test suite lfsck: Failed to find fid

Bug Blocker Resolved Fixed Niu Yawei Maloo MB Tue, 16 Apr 2013 18:51:20 +0000 Thu, 12 Sep 2013 23:58:08 +0000 Tue, 27 Aug 2013 05:34:10 +0000 Lustre 2.4.0 Lustre 2.5.0 Lustre 2.4.1 Lustre 2.5.0 0 9 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">03:53:10:Warning! /dev/mapper/lvm--OSS-P7 is in use. 03:53:10:Warning: skipping journal recovery because doing a read-only filesystem check. </pre> </div></div> <p>The lfsck is doing read-only filesystem check and the FSCK_MAX_ERR should be 4 in this case, but it failed with 1 at the end:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">03:53:14: lfsck : @@@@@@ FAIL: lfsck test 2 - finished with rc=1 </pre> </div></div> <p>Looks the FSCK_MAX_ERR sematics was broken in the fix of <a href="https://jira.whamcloud.com/browse/LU-2571" title="test: lfsck FAIL: lfsck returned 127, should be &lt;= 1" class="issue-link" data-issue-key="LU-2571"><del>LU-2571</del></a> (<a href="http://review.whamcloud.com/#patch,sidebyside,5139,6,lustre/tests/test-framework.sh" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#patch,sidebyside,5139,6,lustre/tests/test-framework.sh</a>), see run_lfsck_remote():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ $rc -le $FSCK_MAX_ERR ] || error <span class="code-quote">"$cmd returned $rc, should be &lt;= $FSCK_MAX_ERR"</span> echo <span class="code-quote">"lfsck finished with rc=$rc"</span> <span class="code-keyword">return</span> $rc </pre> </div></div> <p>I think we'd return 0 when ($rc -le $FSCK_MAX_ERR) but not return $rc.</p> <p>After looking into lfsck.sh, I find we can't return 0 directly because lfsck.sh checks the return value of run_lfsck() to see whether a second run is needed or not. BTW, the directory .lustre is not empty in the current master code, which makes is_empty_fs() always return false. We'd better have a check and fix.</p> <p><a href="http://review.whamcloud.com/6123" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6123</a></p> <p>.lustre/fid I mentioned in last comment was introduced in <a href="https://jira.whamcloud.com/browse/LU-2780" title="Use real inode for .lustre/fid" class="issue-link" data-issue-key="LU-2780"><del>LU-2780</del></a> <a href="http://review.whamcloud.com/#change,5298" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5298</a></p> <p>After discussion with Niu, we found there were several problems in lfsck test, including</p> <p>1) since <a href="https://jira.whamcloud.com/browse/LU-2780" title="Use real inode for .lustre/fid" class="issue-link" data-issue-key="LU-2780"><del>LU-2780</del></a> landed, is_empty_fs() should be changed;</p> <p>2) the test with the patch for 1) showed lfsck failed on empty fs; the output like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfsck: pass4 finished lfsck: exit with 10 unfixed errors lfsck finished with rc=2 removed `/tmp/mdsdb' removed `/tmp/mdsdb.mdshdr' removed `/tmp/ostdb-0' removed `/tmp/ostdb-1' lfsck : @@@@@@ FAIL: lfsck test 2 - finished with rc=2 Trace dump: = /root/master/lustre/tests/test-framework.sh:4022:error_noexit() = /root/master/lustre/tests/test-framework.sh:4045:error() = lfsck.sh:283:main() </pre> </div></div> <p>3) the meaning of FSCK_MAX_ERR is not clear. IMO, when fs is not empty, we only do a read-only filesystem check, so FSCK_MAX_ERR=4(File system errors left uncorrected) means we don't need to do a second check?</p> <p>Looks the lfsck failed to fix some missing objects inode due to the "Failed to find fid <span class="error">&#91;0x2000013a1:0xda11:0x0&#93;</span>: DB_NOTFOUND: No matching key/data pair found", however, before the fix of <a href="https://jira.whamcloud.com/browse/LU-2571" title="test: lfsck FAIL: lfsck returned 127, should be &lt;= 1" class="issue-link" data-issue-key="LU-2571"><del>LU-2571</del></a>, the test will still pass because the run_lfsck() always return 0, thus lfsck.sh won't invoke the second run_lfsck() to verify if the problems are really fixed.</p> <p>The fix of <a href="https://jira.whamcloud.com/browse/LU-2571" title="test: lfsck FAIL: lfsck returned 127, should be &lt;= 1" class="issue-link" data-issue-key="LU-2571"><del>LU-2571</del></a> fixed the script problem in run_lfsck(), we do verify now, and all lfsck tests should fail (except the case of running lfsck on a clean fs).</p> <p>I'll look closer on the problem of "Failed to find fid ...", seems it's a long-standing problem since 2.1.</p> <p>Lustre Branch: b2_4<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/27/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/27/</a></p> <p>The latest test results showed that lfsck test still failed:<br/> <a href="https://maloo.whamcloud.com/test_sets/9743f75a-fd7d-11e2-9fdb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9743f75a-fd7d-11e2-9fdb-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/16153bd0-fdaa-11e2-9fd5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/16153bd0-fdaa-11e2-9fd5-52540035b04c</a></p> <p>It has never passed on Lustre b2_4 branch.</p> <p>few more problems are found when investigate on this ticket:</p> <ul class="alternate" type="square"> <li>ll_lov_setea() overflowed flags, I opened <a href="https://jira.whamcloud.com/browse/LU-3744" title="wrong flags type in ll_lov_setea()" class="issue-link" data-issue-key="LU-3744"><del>LU-3744</del></a> for it;</li> <li>typo in is_empty_fs();</li> <li>in lfsck.sh, sync should be preformced when necessary to make sure data flushed back;</li> <li>if the first run of lfsck fixed some errors, the second run of lfsck will not return clean (0) as expected, it'll always return 1 (some errors fixed) instead, the reason of this is unknown yet (and it exists from day one), but I think we'd go back to the old lfsck to make it pass at this moment, and fix it later;</li> </ul> <p>Niu, how much effort do you think it is to fix these problems? It concerns me that we need to spend time to fix the old lfsck, when the new LFSCK is going to replace it soon. Also, if this has been broken since 2.1, I don't think users could be depending on it very heavily.</p> <p>Finally, I'm also concerned that if the old lfsck is run on a DNE filesystem with multiple MDTs and/or OSTs with FID-on-OST enabled, it is going to do completely the wrong thing, possibly deleting a large number of "unused" objects that are not referenced by MDT0000.</p> <p>At a minimum, a check should be added to old lfsck to refuse to run if it finds signs of DNE (e.g. O/seq, seq &gt; 2) on the OSTs.</p> <p>Hi, Andreas</p> <p>I don't have any idea on these two problems ("Fail to find FID" &amp; "always return 1 on second lfsck run") so far, to fix them, I think I probably need to read most of the lfsck code, that's not a small task.</p> <p>Given that it'll be replaced with new LFSCK soon, and no customer complained about these two problems, I tend to think we'd leave them behind. Maybe we need only to fix the problem on DNE mentioned by you?</p> <p>I certainly think that it makes sense to create new LU tickets for the remaining issues and consider the priority of those separately</p> <p>I created <a href="https://jira.whamcloud.com/browse/LU-3837" title="problem of run old lfsck on DNE" class="issue-link" data-issue-key="LU-3837"><del>LU-3837</del></a> for the old lfsck on DNE problem. I think we'd keep this ticket open (the "Failed to find fid" problem), but lower the priority, because lfsck test won't fail for this error message.</p> <p>I created <a href="https://jira.whamcloud.com/browse/LU-3838" title="lfsck: Failed to find fid" class="issue-link" data-issue-key="LU-3838"><del>LU-3838</del></a> for the two remaining issues, this ticket can be closed.</p> <p>patch landed on b2_4 &amp; master.</p> Related LU-3838 LU-3367 Development Rank 1|hzvo9j: Rank (Obsolete) 7749 Severity