https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9891 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/9e03462a-83bd-11e7-9952-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e03462a-83bd-11e7-9952-5254006e85c2</a>.</p> <p>The sub-test test_7 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15995648 &gt; 15995136 + logsize 400 </pre> </div></div> <p>Info required for matching: replay-ost-single 7</p> LU-9891

replay-ost-single test_7: 15995648 > 15995136 + logsize 400

Bug Minor Resolved Duplicate James Nunez Bob Glossman Fri, 18 Aug 2017 12:39:51 +0000 Thu, 12 Apr 2018 17:42:40 +0000 Thu, 12 Apr 2018 17:42:11 +0000 Lustre 2.11.0 0 12 <p>On master, replay-ost-single test_7 failed, with this error, once in June and has so far failed six times since August 4 and is only failing in for ZFS. </p> <p>The amount of space released by ZFS is not 100% deterministic due to COW and snapshots, as seen by patches in <a href="https://jira.whamcloud.com/browse/LU-2903" title="replay-ost-single test_6 and test_7 failure" class="issue-link" data-issue-key="LU-2903"><del>LU-2903</del></a>. In this case, there is 512KB of space not released by ZFS, but I don't think this is a significant problem. This needs a patch to change fs_log_size() to return 512 for ZFS. </p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28682" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28682</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9891" title="replay-ost-single test_7: 15995648 &gt; 15995136 + logsize 400" class="issue-link" data-issue-key="LU-9891"><del>LU-9891</del></a> tests: Increase space not released for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc82c481d34ead960a3161ed517c6407701990b4</p> <p>Note: replay-ost-single test_6 is also failing with this error message. This test started failing with this error message on August 28, 2017.</p> <p>Logs for some of these failures are at<br/> <a href="https://testing.hpdd.intel.com/test_sets/8cf5f5c6-8c32-11e7-b50a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8cf5f5c6-8c32-11e7-b50a-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/72ae91b2-8dbf-11e7-b5c1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/72ae91b2-8dbf-11e7-b5c1-5254006e85c2</a></p> <p>another on b2_10:<br/> <a href="https://testing.hpdd.intel.com/test_sets/884f3d08-9361-11e7-b722-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/884f3d08-9361-11e7-b722-5254006e85c2</a></p> <p>Andreas - Looking at recent failures for replay-ost-single tests 6 and 7, we see that the space not released by ZFS is between 512 - 1408 KB. For example, see the following logs for ZFS not releasing 1408 KB <a href="https://testing.hpdd.intel.com/test_sets/27a01d7e-91e5-11e7-b67f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/27a01d7e-91e5-11e7-b67f-5254006e85c2</a> and <a href="https://testing.hpdd.intel.com/test_sets/dc64c2f0-8d03-11e7-b50a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dc64c2f0-8d03-11e7-b50a-5254006e85c2</a>. </p> <p>Is ZFS not releasing over 1MB expected and not a problem? If so, then maybe we should increase the return value of fs_log_size() for ZFS to be 1500?</p> <p>The problem with ZFS is that it internally saves 3-4 snapshots of the filesystem for recovery purposes, in case of disk corruption after a crash (e.g. Disks that lie about sync actually getting everything safe in disk). This means that space that is deleted may take 3-4 ZFS transaction commits before deleted files actually release thei space. </p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/edce0d34-95bf-11e7-b75f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/edce0d34-95bf-11e7-b75f-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28682/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28682/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9891" title="replay-ost-single test_7: 15995648 &gt; 15995136 + logsize 400" class="issue-link" data-issue-key="LU-9891"><del>LU-9891</del></a> tests: Increase space not released for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6faae60be03020f1e8b032c1eb93829f70d1e08d</p> <p>Landed for 2.11</p> <p>Hi,</p> <p>I think I hit this issue on master after this patch landed:<br/> <a href="https://testing.hpdd.intel.com/test_sets/82efaa38-98a1-11e7-ba20-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/82efaa38-98a1-11e7-ba20-5254006e85c2</a></p> <p>Correct me if I am wrong.</p> <p>This issue is still seen during testing. Unfortunately, the ZFS log size can be greater than the size of the buffer; see Andreas' comments above. Maybe for ZFS this error should be ignored or the buffer size be increased (a lot)?</p> <p>another on b2_10:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3b6205a0-9e99-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3b6205a0-9e99-11e7-b778-5254006e85c2</a></p> <p>Hit on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/0d996310-b46b-11e7-a282-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0d996310-b46b-11e7-a282-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9f0eb4ca-cbeb-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f0eb4ca-cbeb-11e7-a066-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/21716306-d1b2-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/21716306-d1b2-11e7-a066-52540065bddc</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/abab0d28-f42c-11e7-a169-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/abab0d28-f42c-11e7-a169-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/84bf9c66-f75f-11e7-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/84bf9c66-f75f-11e7-a6ad-52540065bddc</a></p> <p>Patch <a href="https://review.whamcloud.com/#/c/30916/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30916/</a> should fix this failure.</p> <p>James, any chance you could update that patch per review comments?</p> <p><a href="https://testing.hpdd.intel.com/test_sets/0a1d113e-ffd2-11e7-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0a1d113e-ffd2-11e7-a10a-52540065bddc</a></p> <p>+1 on b2_10</p> <p><a href="https://testing.hpdd.intel.com/test_sets/b586c42e-18f1-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b586c42e-18f1-11e8-a7cd-52540065bddc</a></p> <p>another on b2_10:<br/> <a href="https://testing.hpdd.intel.com/test_sets/265c0004-1cd7-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/265c0004-1cd7-11e8-a6ad-52540065bddc</a></p> <p>The last time replay-ost-single test 7 failed testing was on 2018-01-24. It looks like this issue is fixed on the master (pre-2.11) branch.</p> <p>Believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-5761" title="replay-single test_89: @@@@@@ FAIL: 2560 blocks leaked" class="issue-link" data-issue-key="LU-5761"><del>LU-5761</del></a></p> Duplicate LU-5761 LU-3144 Related LU-10052 LU-8672 LU-10352 Development Rank 1|hzzio7: Rank (Obsolete) 9223372036854775807 Severity