Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-3455] Test failure replay-single, test_20b: after 23936 > before 9600 https://jira.whamcloud.com/browse/LU-3455 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run: <br/> <a href="http://maloo.whamcloud.com/test_sets/27d96f00-bf41-11e2-88e0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/27d96f00-bf41-11e2-88e0-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/96ebdda0-c483-11e2-bc10-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/96ebdda0-c483-11e2-bc10-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/7a22e7fe-c762-11e2-ae6b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7a22e7fe-c762-11e2-ae6b-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/911e66d4-c520-11e2-9b87-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/911e66d4-c520-11e2-9b87-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/e4570cf4-c4d3-11e2-ac71-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e4570cf4-c4d3-11e2-ac71-52540035b04c</a></p> <p>The sub-test test_20b failed with the following error:</p> <blockquote> <p>after 23936 &gt; before 9600</p></blockquote> <p>Info required for matching: replay-single 20b</p> LU-3455 Test failure replay-single, test_20b: after 23936 > before 9600 Bug Critical Resolved Fixed Hongchao Zhang Maloo Tue, 11 Jun 2013 18:45:20 +0000 Thu, 26 Feb 2015 03:34:13 +0000 Mon, 12 Jan 2015 15:42:42 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.2 Lustre 2.5.3 Lustre 2.7.0 Lustre 2.5.4 0 13 <p><a href="https://maloo.whamcloud.com/test_sets/f84d0b8a-f46e-11e2-b8a2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f84d0b8a-f46e-11e2-b8a2-52540035b04c</a></p> <p>This is still failing quite a bit.</p> <p>This fails quite a bit more often on review-dne-zfs, but there are still review-zfs failures as well. This might be because the amount of extra data written for DNE is higher than for non-DNE systems? Should the error margin be scaled by MDSCOUNT in the test?</p> <p>MDS debug_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00080000:0.0:1381566255.035796:0:14198:0:(osp_dev.c:1039:osp_import_event()) got inactive 00000100:00080000:0.0:1381566255.035798:0:14198:0:(pinger.c:433:ptlrpc_pinger_del_import()) removing pingable import lustre-MDT0000-mdtlov_UUID-&gt;lustre-OST0000_UUID 00000100:00080000:0.0:1381566255.035811:0:14198:0:(import.c:1484:ptlrpc_disconnect_import()) ffff8800298b4800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:0.0:1381566255.035819:0:14198:0:(client.c:1469:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc umount:lustre-MDT0000-mdtlov_UUID:14198:1448660421728700:10.10.16.157@tcp:9 00000100:00100000:0.0:1381566255.035854:0:14198:0:(client.c:2117:ptlrpc_set_wait()) set ffff8800571a9bc0 going to sleep for 5 seconds 00000004:00080000:0.0:1381566255.035961:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880056eb2800/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.035969:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728420:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.035974:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880025f2ec00/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.035977:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728428:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.035980:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880025f2e000/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.035983:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728436:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.035986:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff88006c467800/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.035989:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728444:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.035997:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880056eb2c00/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.036000:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728416:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.036004:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880056eb2000/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.036007:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728424:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.036011:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff880025f2e400/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.036014:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728432:10.10.16.157@tcp:6 00000004:00080000:0.0:1381566255.036018:0:3486:0:(osp_sync.c:348:osp_sync_interpret()) reply req ffff88006c467c00/1, rc -5, transno 0 00000100:00100000:0.0:1381566255.036020:0:3486:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:lustre-MDT0000-mdtlov_UUID:3486:1448660421728440:10.10.16.157@tcp:6 00000100:00100000:0.0:1381566255.036050:0:13350:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc osp-pre-0:lustre-MDT0000-mdtlov_UUID:13350:1448660421728660:10.10.16.157@tcp:5 00000004:00020000:0.0:1381566255.036055:0:13350:0:(osp_precreate.c:481:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 00000004:00020000:0.0:1381566255.038361:0:13350:0:(osp_precreate.c:986:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 </pre> </div></div> <p>WAG: Might this be related to <a href="http://review.whamcloud.com/6988" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6988</a> (from <a href="https://jira.whamcloud.com/browse/LU-3230" title="conf-sanity fails to start run: umount of OST fails" class="issue-link" data-issue-key="LU-3230"><del>LU-3230</del></a>)? That patch would cause the EIO to propogate and thus fail the precreates and cause the size mismatch?</p> <p>Update: This bug predates landing of accused patch, and is thus not the culprit.</p> <p>The first occurrence of this bug is actually on <a href="http://review.whamcloud.com/5666" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5666</a> (<a href="https://jira.whamcloud.com/browse/LU-2342" title="replay-single test_20b: @@@@@@ FAIL: after 6912 &gt; before 6784" class="issue-link" data-issue-key="LU-2342"><del>LU-2342</del></a>) which re-enabled this test for ZFS.</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/709b23f6-6e9f-11e3-b713-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/709b23f6-6e9f-11e3-b713-52540035b04c</a></p> <p>review-zfs on master:<br/> <a href="https://maloo.whamcloud.com/test_sets/1b83d22e-90b8-11e3-9767-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1b83d22e-90b8-11e3-9767-52540035b04c</a></p> <p>More instance on master branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/d4bf6952-9792-11e3-877b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d4bf6952-9792-11e3-877b-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/63/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/63/</a><br/> TEST_GROUP=failover<br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/1a08ad42-f374-11e3-86ca-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1a08ad42-f374-11e3-86ca-52540035b04c</a></p> <p>Hongchao</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>the debug patch is tracked at <a href="http://review.whamcloud.com/#/c/11471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11471/</a></p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/84/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/84/</a><br/> TEST_GROUP=failover<br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://testing.hpdd.intel.com/test_sets/e169ad78-2ab5-11e4-aec8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e169ad78-2ab5-11e4-aec8-5254006e85c2</a></p> <p>this bug is related to the ZFS, which only release occupied disk space once DMUs are committed?</p> <p>the tests of 20b of replay_sinlge were all passed if waiting some time after "wait_delete_completed_mds", and all failed without change<br/> (wait 5 seconds) <a href="https://testing.hpdd.intel.com/test_sessions/d79c9e64-2a69-11e4-8657-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d79c9e64-2a69-11e4-8657-5254006e85c2</a><br/> (no wait) <a href="https://testing.hpdd.intel.com/test_sessions/15b6d73a-2c4c-11e4-8362-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/15b6d73a-2c4c-11e4-8362-5254006e85c2</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">wait_delete_completed_mds() { ... # sync MDS transactions do_nodes $mds2sync <span class="code-quote">"$LCTL set_param -n osd*.*MD*.force_sync 1"</span> # wait till all changes are sent and commmitted by OSTs # <span class="code-keyword">for</span> ldiskfs space is released upon execution, but DMU # <span class="code-keyword">do</span> <span class="code-keyword">this</span> upon commit local WAIT=0 <span class="code-keyword">while</span> [[ $WAIT -ne $MAX_WAIT ]]; <span class="code-keyword">do</span> changes=$(do_nodes $mds2sync \ <span class="code-quote">"$LCTL get_param -n osc.*MDT*.sync_*"</span> | calc_sum) #echo <span class="code-quote">"$node: $changes changes on all"</span> <span class="code-keyword">if</span> [[ $changes -eq 0 ]]; then etime=$(date +%s) #echo <span class="code-quote">"delete took $((etime - stime)) seconds"</span> <span class="code-keyword">return</span> fi sleep 1 WAIT=$(( WAIT + 1)) done ... } </pre> </div></div> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/86/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/86/</a> (2.5.3 RC1)<br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://testing.hpdd.intel.com/test_sets/158df844-311d-11e4-b503-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/158df844-311d-11e4-b503-5254006e85c2</a></p> <p>the initial patch is tracked at <a href="http://review.whamcloud.com/#/c/11767/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11767/</a></p> <p>the patch has been updated to sync the underlying OSD if some client is being evicted.</p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d89c27d6-5b1e-11e4-9c62-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d89c27d6-5b1e-11e4-9c62-5254006e85c2</a></p> <p>This might be related to <a href="https://jira.whamcloud.com/browse/LU-4265" title="replay-ost-single test_6: space grew after dd (or didn&#39;t change)" class="issue-link" data-issue-key="LU-4265"><del>LU-4265</del></a>. Free space accounting with ZFS can be quite tricky as sometimes frees can be delayed for quite a while.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/11767/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11767/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3455" title="Test failure replay-single, test_20b: after 23936 &gt; before 9600" class="issue-link" data-issue-key="LU-3455"><del>LU-3455</del></a> mdt: sync when evicting client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fabd01968d0fac83e4855da5a5dfe8f68c038ceb</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13231" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13231</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3455" title="Test failure replay-single, test_20b: after 23936 &gt; before 9600" class="issue-link" data-issue-key="LU-3455"><del>LU-3455</del></a> mdt: sync when evicting client<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: aa79c4974653d101be9d03200914b023970a7d41</p> <p>Patch landed to Master. b2_5 patch is being tracked externally.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13231/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13231/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3455" title="Test failure replay-single, test_20b: after 23936 &gt; before 9600" class="issue-link" data-issue-key="LU-3455"><del>LU-3455</del></a> mdt: sync when evicting client<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: fd7d369b7a513fd0b3123d16b8329a0cfc97875c</p> <p>We still hit it on b2_5 with ZFS backend:<br/> <a href="https://testing.hpdd.intel.com/test_sets/05e4fbe4-bd23-11e4-a946-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/05e4fbe4-bd23-11e4-a946-5254006e85c2</a></p> Related LU-3626 LU-4051 Development Rank 1|hzvt1z: Rank (Obsolete) 8637 Severity