https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1966 Lustre <p>This issue was created by maloo for yujian &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/0d43f964-fff5-11e1-9f3c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0d43f964-fff5-11e1-9f3c-52540035b04c</a>.</p> <p>The sub-test test_6 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-ost-single test 6: Fail OST before obd_destroy == 19:17:36 (1347761856) Waiting for orphan cleanup... CMD: client-28vm4 /usr/sbin/lctl get_param -n obdfilter.*.mds_sync Waiting for destroy to be done... Waiting 0 secs for destroys to be done. Waiting 1 secs for destroys to be done. Waiting 2 secs for destroys to be done. Waiting 3 secs for destroys to be done. Waiting 4 secs for destroys to be done. Destroys weren't done in 5 sec. replay-ost-single test_6: @@@@@@ FAIL: test_6 failed with 5 </pre> </div></div> <p>Info required for matching: replay-ost-single 6</p> LU-1966

Test failure on test suite replay-ost-single, subtest test_6: Destroys weren't done in 5 sec

Bug Minor Resolved Fixed Zhenyu Xu Maloo Mon, 17 Sep 2012 09:28:34 +0000 Tue, 5 Mar 2013 15:14:46 +0000 Tue, 5 Mar 2013 15:14:46 +0000 Lustre 2.3.0 0 5 <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_3/19" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/19</a><br/> Test group: failover</p> <p>replay-ost-single test 7 also failed with the same issue: <a href="https://maloo.whamcloud.com/test_sets/0d43f964-fff5-11e1-9f3c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0d43f964-fff5-11e1-9f3c-52540035b04c</a></p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Peter</p> <p>looks like previous tests residue interference.</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>OSS debug log</b></div><div class="panelContent"> <p><font color="blue">line:9295</font> 00000100:00100000:0.0:1347761823.542774:0:2427:0:(service.c:1786:ptlrpc_server_handle_req_in()) got req x1413227639926489<br/> <font color="blue">line:9296</font> 00000100:00080000:0.0:1347761823.542777:0:2427:0:(service.c:1000:ptlrpc_update_export_timer()) updating export 3c3577cd-0751-66ec-7591-46dd3a204f77 at 1347761823 exp ffff88007c77a800<br/> <font color="blue">line:9297</font> 00000100:00100000:0.0:1347761823.542788:0:2427:0:(service.c:1961:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io00_009:3c3577cd-0751-66ec-7591-46dd3a204f77+996:3262:x1413227639926489:12345-10.10.4.172@tcp:6<br/> ...<br/> <font color="blue">line:10263</font> 00000001:02000400:0.0:1347761856.433208:0:4282:0:(debug.c:445:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 6: Fail OST before obd_destroy == 19:17:36 &#40;1347761856&#41;</p> <p>... <font color="red">pid 2427 leaves no footprint in OSS log thereafter, even after test 6 and test 7 fails, OSS log does not show that it processes the OST_DESTROY request </font> </p> </div></div> <p>Status update:</p> <p>Booked one toro node (only one is available), ran "bash /usr/lib64/lustre/tests/auster -rsv replay-ost-single", and it finished w/o error.</p> <p>Hi Bobi,<br/> If you wanna reproduce or debug this issue, you can upload a patch to Gerrit with the following test parameters:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: fortestonly envdefinitions=SLOW=yes \ clientcount=4 osscount=2 mdscount=2 austeroptions=-R \ failover=true useiscsi=true testlist=replay-ost-single </pre> </div></div> <p><a href="http://review.whamcloud.com/4067" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4067</a> just for issue reproduction.</p> <p>Here is a link to the historical Maloo reports for replay-ost-single test 6 in <b>failover</b> test group:<br/> <a href="http://tinyurl.com/8s62nfj" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/8s62nfj</a></p> <p>As we can see, the last successful run for this test is on 2012-09-07, and the test has been occurring since 2012-09-13.</p> <p>I checked the patch landed between 09/07 and 09/13, I think nothing about OSS change during that period could cause this issue, I tend to think it's test issue.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/23" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/23</a><br/> FAILURE_MODE=HARD</p> <p>autotest run failed: <a href="https://maloo.whamcloud.com/test_sets/b5c9fc00-0694-11e2-9b17-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b5c9fc00-0694-11e2-9b17-52540035b04c</a><br/> manual run passed: <a href="https://maloo.whamcloud.com/test_sets/52192992-0716-11e2-ac99-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/52192992-0716-11e2-ac99-52540035b04c</a></p> <p>I think that there is enough evidence to suggest that this is purely a testing issue. We should still continue to work on resolving this and include a fix for an RC2 if one is ready and we need one, but it would not warrant holding the release on its own merits</p> <p>Lustre Tag: v2_3_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/32" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/32</a><br/> Distro/Arch: RHEL6.3/x86_64<br/> Test Group: failover</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/c1790860-1509-11e2-9adb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c1790860-1509-11e2-9adb-52540035b04c</a></p> Duplicate LU-2620 Development Rank 1|hzv30v: Rank (Obsolete) 3989 Severity