https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2012 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/3e022626-dd35-11e1-a041-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3e022626-dd35-11e1-a041-52540035b04c</a>.</p> <p>The sub-test test_14b failed with the following error:</p> <blockquote> <p>after 846984 &gt; before 846980</p></blockquote> <p>This was with LDiskFS, so not ORI-396.</p> <p>Info required for matching: replay-dual 14b</p> LU-2012

replay-dual test_14b: after 846984 > before 846980

Bug Minor Reopened Unresolved Mikhail Pershin Maloo always_except ldiskfs zfs Tue, 7 Aug 2012 00:40:00 +0000 Mon, 16 May 2022 22:41:07 +0000 Lustre 2.3.0 Lustre 2.4.0 0 7 <p>This test should be disabled since we disabled gap handling for now</p> <p><a href="https://maloo.whamcloud.com/test_sets/044f5114-fa05-11e1-8ea7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/044f5114-fa05-11e1-8ea7-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/189f38d8-049a-11e2-bfd4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/189f38d8-049a-11e2-bfd4-52540035b04c</a></p> <p>This was master with OFD and LDiskFS OSTs.</p> <p><a href="https://maloo.whamcloud.com/test_sets/0e1568ba-0611-11e2-9b17-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0e1568ba-0611-11e2-9b17-52540035b04c</a></p> <p>This was master with OFD and LDiskFS OSTs.</p> <p><a href="https://maloo.whamcloud.com/test_sets/0b57c26e-11ae-11e2-9408-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0b57c26e-11ae-11e2-9408-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/e7e7add4-1203-11e2-a663-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e7e7add4-1203-11e2-a663-52540035b04c</a></p> <p>I hit this with ldiskfs in local testing as well:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>replay-dual test_14b: @@@@@@ FAIL: after 76856 &gt; before 76852 </pre> </div></div> <p>but it isn't easy to determine if the small number of extra blocks is a clear indication of test failure, or some other usage in the background (e.g. llog records, etc). The orphan files should be created with a noticeable amount of data, so that if they fail to be deleted it will be obvious.</p> <p>Submitted patch to disable test_14b until orphan handling is fixed: <a href="http://review.whamcloud.com/4237" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4237</a></p> <p>The test itself is improved from the previous version, since it writes a large orphan file and it allows for some small margin of error in the df output due to allocated blocks (logs, OI, etc).</p> <p>Lustre b2_1 client build: <a href="http://build.whamcloud.com/job/lustre-b2_1/176" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/176</a><br/> Lustre master server build: <a href="http://build.whamcloud.com/job/lustre-master/1269" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1269</a><br/> Distro/Arch: RHEL6.3/x86_64</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/a19d48ee-7d78-11e2-85d0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a19d48ee-7d78-11e2-85d0-52540035b04c</a></p> <p>The same failure also occurred on Lustre b2_1 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/198fd69c-7e6d-11e2-8b88-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/198fd69c-7e6d-11e2-8b88-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/1a65e4fc-7d65-11e2-ac80-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1a65e4fc-7d65-11e2-ac80-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/dca53842-7c48-11e2-897d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dca53842-7c48-11e2-897d-52540035b04c</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/181" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/181</a><br/> The issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/511b9cd0-8251-11e2-8172-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/511b9cd0-8251-11e2-8172-52540035b04c</a></p> <p>Hi Andreas,</p> <p>Does the fix in <a href="http://review.whamcloud.com/4237" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4237</a> need to be ported to Lustre b2_1 branch?</p> <p>The failure occurs constantly on Lustre b2_1 branch: <a href="https://maloo.whamcloud.com/test_sets/38643f8c-826d-11e2-ba47-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/38643f8c-826d-11e2-ba47-52540035b04c</a></p> <p>Patch for Lustre b2_1 branch: <a href="http://review.whamcloud.com/5571" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5571</a></p> <p>Assign to Yu Jian for follow-up patch to remove test_14b from ALWAYS_EXCEPT to see if the changes to the test itself (larger test file and a small allocation allowance) are enough to allow it to pass regularly.</p> <p>Lustre Branch: master</p> <p>After removing replay-dual 14b from ALWAYS_EXCEPT list, the test still failed:<br/> <a href="https://maloo.whamcloud.com/test_sets/aeeec35c-87d6-11e2-961a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/aeeec35c-87d6-11e2-961a-52540035b04c</a></p> <p>This failed test is clearly caused by a defect in the code, since the size difference is 5120kB = 5MB, which is the size of the file that should have been deleted by orphan recovery. This is definitely not a case of some other blocks being allocated during testing.</p> <p>I think Niu was working on a patch related to orphan recovery for 2.1, and Mike was also working on this. Perhaps they already know what the problem is here.</p> <p>In fact we have just disabled gap handling that is why orphans may stay on OST now. Last time we discussed that the solution was 'run lfsck'<br/> Nevertheless I had some ideas how to handle gaps and destroy orphans, so if lfsck is not the case now we can reconsider to fix that in other way.</p> <p>The LFSCK MDS-OSS checking will not be available until at least 2.6, so if we can get a solution in the meantime that would be good. Note that there was a bug that Niu was working on where non-orphan files were being deleted, so we don't want to reintroduce this. </p> <p>yes, that is exactly why we disabled it. I will refresh my memory about alternative gap handling then</p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)<br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)</p> <p>replay-dual test 14b hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/43411d98-14fe-11e3-9828-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/43411d98-14fe-11e3-9828-52540035b04c</a></p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)<br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_4/69/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/69/</a> (2.4.2 RC1)</p> <p>replay-dual test 14b hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/9369eef2-6860-11e3-a16f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9369eef2-6860-11e3-a16f-52540035b04c</a></p> <p>This is in <tt>ALWAYS_EXCEPT</tt> so that is the reason it has not failed recently.</p> Duplicate LU-10052 LU-10608 LU-3214 Related Development Rank 1|hzuxfz: Rank (Obsolete) 3080 Severity