https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6374 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/77b52408-ccc1-11e4-a8ca-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/77b52408-ccc1-11e4-a8ca-5254006e85c2</a>.</p> <p>This looks similar to <a href="https://jira.whamcloud.com/browse/LU-3455" title="Test failure replay-single, test_20b: after 23936 &gt; before 9600" class="issue-link" data-issue-key="LU-3455"><del>LU-3455</del></a> but that is marked Resolved.<br/> It may be zfs only. This instance was seen during review-zfs on master.</p> <p>The sub-test test_20b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>after 44416 &gt; before 6528 </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: replay-single 20b</p> LU-6374

replay-single test_20b: after 44416 > before 6528

Bug Minor Resolved Fixed Niu Yawei Maloo Tue, 17 Mar 2015 17:52:57 +0000 Wed, 21 Dec 2016 05:30:38 +0000 Wed, 21 Dec 2016 05:30:38 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.10.0 0 9 <p>Another on 2.7.55+ <a href="https://testing.hpdd.intel.com/test_sets/4323393a-250b-11e5-8427-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4323393a-250b-11e5-8427-5254006e85c2</a>.</p> <p>Also <a href="https://testing.hpdd.intel.com/test_sets/08cba93c-3db3-11e5-9e7f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/08cba93c-3db3-11e5-9e7f-5254006e85c2</a> on master.</p> <p>More instances on master, all ZFS:<br/> 2015-11-04 21:19:27 - <a href="https://testing.hpdd.intel.com/test_sets/87ec5f46-8376-11e5-8df7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/87ec5f46-8376-11e5-8df7-5254006e85c2</a><br/> 2015-11-21 15:51:23 - <a href="https://testing.hpdd.intel.com/test_sets/bf7d6960-90a4-11e5-b9af-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bf7d6960-90a4-11e5-b9af-5254006e85c2</a><br/> 2015-12-09 00:09:30 - <a href="https://testing.hpdd.intel.com/test_sets/50aeefd8-9e44-11e5-86f6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/50aeefd8-9e44-11e5-86f6-5254006e85c2</a><br/> 2016-02-21 14:54:24 - <a href="https://testing.hpdd.intel.com/test_sets/c7a4e68a-d8f3-11e5-83e2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c7a4e68a-d8f3-11e5-83e2-5254006e85c2</a></p> <p>More instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/2a69c5be-aaba-11e5-9fbe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2a69c5be-aaba-11e5-9fbe-5254006e85c2</a></p> <p>Hit on master: <a href="https://testing.hpdd.intel.com/test_sets/ed824568-a662-11e6-a6e7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ed824568-a662-11e6-a6e7-5254006e85c2</a></p> <p>For zfs, we need wait for commit to release space, but looks wait_delete_completed_mds() didn't wait at all:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">wait_delete_completed_mds() { local MAX_WAIT=${1:-20} # <span class="code-keyword">for</span> ZFS, waiting more time <span class="code-keyword">for</span> DMUs to be committed local ZFS_WAIT=${2:-5} local mds2sync="" local stime=$(date +%s) local etime local node local changes # find MDS with pending deletions <span class="code-keyword">for</span> node in $(mdts_nodes); <span class="code-keyword">do</span> changes=$(do_node $node <span class="code-quote">"$LCTL get_param -n osc.*MDT*.sync_*"</span> \ 2&gt;/dev/<span class="code-keyword">null</span> | calc_sum) <span class="code-keyword">if</span> [[ $changes -eq 0 ]]; then <span class="code-keyword">continue</span> fi mds2sync=<span class="code-quote">"$mds2sync $node"</span> done <span class="code-keyword">if</span> [ -z <span class="code-quote">"$mds2sync"</span> ]; then <span class="code-keyword">return</span> &lt;------------- before <span class="code-keyword">this</span> <span class="code-keyword">return</span>, we need to wait <span class="code-keyword">for</span> zfs commit fi </pre> </div></div> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23688" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23688</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6374" title="replay-single test_20b: after 44416 &gt; before 6528" class="issue-link" data-issue-key="LU-6374"><del>LU-6374</del></a> tests: wait for zfs commit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 19ad6284c99469aa193760228b6830e5d221a553</p> <p>The failure can be easily reproduced locally with zfs backend, and with above patch applied, I can't reproduce it anymore.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23688/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23688/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6374" title="replay-single test_20b: after 44416 &gt; before 6528" class="issue-link" data-issue-key="LU-6374"><del>LU-6374</del></a> tests: wait for zfs commit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6f680d9eef683b83b478ff2aaf281d15f7c78fa2</p> <p>Landed in Lustre 2.10.0</p> Development Rank 1|hzx8pb: Rank (Obsolete) 9223372036854775807 Severity