https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4265 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/627ba06c-48b0-11e3-bdb5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/627ba06c-48b0-11e3-bdb5-52540035b04c</a><br/> <a href="http://maloo.whamcloud.com/test_sets/600e6472-4f68-11e3-84d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/600e6472-4f68-11e3-84d3-52540035b04c</a></p> <p>The sub-test test_6 failed with the following error:</p> <blockquote> <p>space grew after dd: before:77312128 after_dd:77312128</p></blockquote> <p>Info required for matching: replay-ost-single 6</p> LU-4265

replay-ost-single test_6: space grew after dd (or didn't change)

Bug Major Resolved Cannot Reproduce WC Triage Maloo zfs Mon, 18 Nov 2013 16:06:43 +0000 Mon, 29 Apr 2019 17:55:03 +0000 Mon, 29 Apr 2019 17:55:02 +0000 Lustre 2.6.0 Lustre 2.5.3 Lustre 2.9.0 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.1 0 10 <p>While verifying patches <a href="http://review.whamcloud.com/11541" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11541</a>, <a href="http://review.whamcloud.com/11411" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11411</a>, <a href="http://review.whamcloud.com/9318" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9318</a> on Lustre b2_5 branch with FSTYPE=zfs, the same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/160d2586-31e7-11e4-92e0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/160d2586-31e7-11e4-92e0-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/228a2e1e-27b8-11e4-893b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/228a2e1e-27b8-11e4-893b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/cb2bd86a-9a50-11e3-965c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cb2bd86a-9a50-11e3-965c-52540035b04c</a></p> <p>One more instance on Lustre b2_5 branch: <a href="https://testing.hpdd.intel.com/test_sets/cd0bb8ee-44dc-11e4-bb5a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cd0bb8ee-44dc-11e4-bb5a-5254006e85c2</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3e9f4124-45c3-11e4-9397-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3e9f4124-45c3-11e4-9397-5254006e85c2</a></p> <p>Another one:<br/> <a href="https://testing.hpdd.intel.com/test_sets/197aed5c-592e-11e4-8f95-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/197aed5c-592e-11e4-8f95-5254006e85c2</a></p> <p>It appeared to be related to <a href="https://jira.whamcloud.com/browse/LU-3455" title="Test failure replay-single, test_20b: after 23936 &gt; before 9600" class="issue-link" data-issue-key="LU-3455"><del>LU-3455</del></a>. The test_6 tried to get before=$(kbytesfree) after file removal and some wait:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> rm -f $f sync &amp;&amp; sleep 5 &amp;&amp; sync # wait for delete thread # wait till space is returned, following # (( $before &gt; $after_dd)) test counting on that wait_mds_ost_sync || return 4 wait_destroy_complete || return 5 local before=$(kbytesfree) </pre> </div></div> <p>I'd doubt the wait can work reliably with ZFS - sometimes frees can be delayed for a few transaction groups' time. It seemed inherently unreliable to free and wait and get free space for ZFS.</p> <p>More failure instance on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/cdfb95ec-7dd7-11e4-aa98-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cdfb95ec-7dd7-11e4-aa98-5254006e85c2</a></p> <p>Another instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b07629a6-86a6-11e4-b678-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b07629a6-86a6-11e4-b678-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/92cbbd1a-16d1-11e5-8436-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/92cbbd1a-16d1-11e5-8436-5254006e85c2</a></p> <p>Another two failures on master in review-zfs-part-2:<br/> 2015-07-03 07:45:48 - <a href="https://testing.hpdd.intel.com/test_sets/ceba821c-2161-11e5-a979-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ceba821c-2161-11e5-a979-5254006e85c2</a><br/> 2015-07-14 21:23:00 - <a href="https://testing.hpdd.intel.com/test_sets/234cf58c-2a7a-11e5-96c0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/234cf58c-2a7a-11e5-96c0-5254006e85c2</a></p> <p>More failure instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/94e59696-3640-11e5-84a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/94e59696-3640-11e5-84a9-5254006e85c2</a></p> <p>Another failure on master:<br/> 2015-12-26 18:38:36 - <a href="https://testing.hpdd.intel.com/test_sets/cc75f8cc-ac0a-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cc75f8cc-ac0a-11e5-8114-5254006e85c2</a></p> <p>It looks like this test is failing again/still. I’ve gone back to January 2018 and looks at all the times replay-ost-single test 6 failed with the ''space grew after dd:” error and we’ve seen six failures all for ZFS testing:<br/> 2018-08-17 - 2.11.53.69 - <a href="https://testing.whamcloud.com/test_sets/24be3e5c-a273-11e8-8853-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/24be3e5c-a273-11e8-8853-52540065bddc</a><br/> 2019-02-26 – 2.12.0.1 - <a href="https://testing.whamcloud.com/test_sets/60e5eab4-3964-11e9-8f69-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/60e5eab4-3964-11e9-8f69-52540065bddc</a><br/> 2019-04-17 – 2.12.52.85 - <a href="https://testing.whamcloud.com/test_sets/4fd6a78e-6170-11e9-9720-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4fd6a78e-6170-11e9-9720-52540065bddc</a><br/> 2019-04-19 – 2.12.52.89 - <a href="https://testing.whamcloud.com/test_sets/043e5b44-62fd-11e9-aeec-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/043e5b44-62fd-11e9-aeec-52540065bddc</a><br/> 2019-04-23 – 2.12.52.95 - <a href="https://testing.whamcloud.com/test_sets/bf51ebce-65f7-11e9-a6f9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/bf51ebce-65f7-11e9-a6f9-52540065bddc</a><br/> 2019-04-23 – 2.12.1 RC1 - <a href="https://testing.whamcloud.com/test_sets/d53ee48a-665d-11e9-8bb1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d53ee48a-665d-11e9-8bb1-52540065bddc</a></p> <p>Some of the failures look questionable or, possibly, the error message should be modified. For example, when “before” and “after” are the same value:<br/> ''space grew after dd: before:13442048 after_dd:13442048''</p> <p>This stopped being hit in 2015.</p> Related LU-12232 Development Rank 1|hzw9lr: Rank (Obsolete) 11718 Severity