https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1867 Lustre <p>Hit this problem on Maloo test on latest master branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/07148716-fae1-11e1-a03c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/07148716-fae1-11e1-a03c-52540035b04c</a></p> <p>It is similiar with ORI-412 reported on Orion.</p> <p>Test logs of test_89 attached.</p> LU-1867

replay-single test_89: @@@@@@ FAIL: 4 blocks leaked

Bug Critical Resolved Fixed Yang Sheng Xuezhao Liu Mon, 10 Sep 2012 00:02:45 +0000 Tue, 28 Nov 2017 22:08:53 +0000 Thu, 16 Oct 2014 07:15:09 +0000 Lustre 2.4.0 Lustre 2.4.0 0 8 <p>Again <a href="https://maloo.whamcloud.com/test_sets/de889ae4-fb49-11e1-8e05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/de889ae4-fb49-11e1-8e05-52540035b04c</a></p> <p>test_89 <br/> Error: '4 blocks leaked'<br/> Failure Rate: 8.00% of last 100 executions <span class="error">&#91;all branches&#93;</span> </p> <p><a href="https://maloo.whamcloud.com/test_sessions/dff25e82-fc94-11e1-b09c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/dff25e82-fc94-11e1-b09c-52540035b04c</a></p> <p>Yangsheng</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Another one: <a href="https://maloo.whamcloud.com/test_sets/f621c0de-fc95-11e1-b09c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f621c0de-fc95-11e1-b09c-52540035b04c</a></p> <p>Another one: <a href="https://maloo.whamcloud.com/test_sets/b8eb255e-fcca-11e1-b09c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b8eb255e-fcca-11e1-b09c-52540035b04c</a></p> <p>Increasing priority, since this is causing a fairly high failure rate in tests - about 1 of 6 in the last few days.</p> <p>Searching through the Maloo history for this test, it seems the majority of bugs hit were testing USE_OFD=yes, for <a href="https://jira.whamcloud.com/browse/LU-1871" title="MDS oops in mdsrate-create-small.sh: Thread overran stack, or stack corrupted" class="issue-link" data-issue-key="LU-1871"><del>LU-1871</del></a> on 09/08 or so.</p> <p>Failed <a href="https://maloo.whamcloud.com/test_sets/e3948898-040b-11e2-aec7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e3948898-040b-11e2-aec7-52540035b04c</a>, which didn't run with USE_OFD, though it is an ofd related patch.</p> <p><a href="https://maloo.whamcloud.com/test_sets/e0643d4a-0496-11e2-bfd4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e0643d4a-0496-11e2-bfd4-52540035b04c</a></p> <p>This was master with OFD and LDiskFS OSTs.</p> <p>Has any work been done yet to determine why this test is failing, and what needs to be done to fix it?</p> <p>Can <a href="http://review.whamcloud.com/#change,1704" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1704</a> on Orion help to resolve/reduce this issue on master?</p> <p>Liu Xuezhao,</p> <p>Yes, I think the extra wait_delete_completed() should help reduce the failure rate. That change is already included in <a href="http://review.whamcloud.com/2982" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2982</a>, which hopefully could land soon.</p> <p><a href="https://maloo.whamcloud.com/test_sets/abc64ce8-060f-11e2-9b17-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/abc64ce8-060f-11e2-9b17-52540035b04c</a></p> <p>This was master with OFD and LDiskFS OSTs.</p> <p><a href="https://maloo.whamcloud.com/test_sets/a22a10ee-07df-11e2-9e76-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a22a10ee-07df-11e2-9e76-52540035b04c</a></p> <p>let's disable this test until the root cause is understood. the issue looks pretty local and not affecting other tests, functionality.</p> <p>I have doing some investigate as below. This issue caused by config-llog data not sync between mgs &amp; ost. We count free block first as BLOCK1. Then write data to OST...etc. There have some data wrote in OST of config-data on MGS, but not in OST. Then ost umount, And the config-data will sync when ost remount. There we count free block as BLOCK2. So the BLOCK2 - BLOCK1 is the config-data changes. It may or may not cause a new block be allocated(4k). So we encounter this issue very randomly and the leak block always 4k.</p> <p>In this case, the test pass condition should be changed to allow 4 blocks (16kB) difference between BLOCKS2 and BLOCKS1 and still pass, along with a comment explaining this. I guess this doesn't explain the "1536 blocks leaked" problem seen in other test failures.</p> <p>Patch commit to: <a href="http://review.whamcloud.com/#change,4130" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4130</a></p> <p>Patch landed. Close bug.</p> <p>replay-single test_89 is still being skipped on ZFS due to this bug. It looks like the landed patch may resolve the test failure, so a patch to re-enable it should be submitted.</p> <p>re-enable test patch: <a href="http://review.whamcloud.com/12227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12227</a></p> <p>This is still failing for ZFS so the above patch only re-enables test_89 for ldiskfs:<br/> <a href="https://testing.hpdd.intel.com/test_sets/f4f00d1a-4ffe-11e4-8734-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f4f00d1a-4ffe-11e4-8734-5254006e85c2</a></p> <p>I think this ticket is for ldiskfs issue. ZFS has similar issue but it shows more blocks leak and OR-412 is more proper to handle it. So i close this one first.</p> <p>Note that the ORI project is closed and those tickets cannot be used to land patches on master. I opened <a href="https://jira.whamcloud.com/browse/LU-5761" title="replay-single test_89: @@@@@@ FAIL: 2560 blocks leaked" class="issue-link" data-issue-key="LU-5761"><del>LU-5761</del></a> for tracking the ZFS issue. </p> <p>This problem is seeing again at:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/a6ab66ac-d1ad-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a6ab66ac-d1ad-11e7-9c63-52540065bddc</a></p> <p>Jinshan, I strongly suspect that if this ancient issue is being seen again that it is a new issue that needs a new Jira ticket, even if the error message is the same. It is also most likely that the problem is FLR related, since this hasn't been reported in over 3 years. </p> <p>It happened only once in these many tests recently. Let's see how it goes. </p> <p>This failure is being tracked under <a href="https://jira.whamcloud.com/browse/LU-5761" title="replay-single test_89: @@@@@@ FAIL: 2560 blocks leaked" class="issue-link" data-issue-key="LU-5761"><del>LU-5761</del></a>.</p> Duplicate Related LU-5761 Development Rank 1|hzv5gn: Rank (Obsolete) 4419 Severity