https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2620 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/478c299a-5ef8-11e2-b507-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/478c299a-5ef8-11e2-b507-52540035b04c</a>.</p> <p>The sub-test test_6 failed with the following error:</p> <blockquote> <p>test_6 failed with 1</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-ost-single test 6: Fail OST before obd_destroy == 17:27:58 (1358126878) Waiting for orphan cleanup... CMD: client-32vm3 /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed Waiting for local destroys to complete 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.970226 s, 5.4 MB/s /mnt/lustre/d0.replay-ost-single/f.replay-ost-single.6 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 193 0xc1 0 CMD: client-32vm3 lctl set_param fail_loc=0x80000119 fail_loc=0x80000119 before: 12650184 after_dd: 13693644 replay-ost-single test_6: @@@@@@ FAIL: test_6 failed with 1 </pre> </div></div> LU-2620

Failure on test suite replay-ost-single test_6: test_6 failed with 1

Bug Blocker Resolved Fixed Bob Glossman Maloo HB Tue, 15 Jan 2013 14:07:55 +0000 Thu, 5 Sep 2013 07:24:25 +0000 Thu, 7 Mar 2013 18:50:55 +0000 Lustre 2.4.0 Lustre 2.1.4 Lustre 1.8.9 Lustre 2.4.0 Lustre 2.1.5 0 10 <p>I think this problem may have been induced by the <a href="https://jira.whamcloud.com/browse/LU-2494" title="error: get_param: /proc/{fs,sys}/{lnet,lustre}/obdfilter/*/mds_sync: Found no match" class="issue-link" data-issue-key="LU-2494"><del>LU-2494</del></a> landing of <a href="http://review.whamcloud.com/4885" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4885</a>.</p> <p>So far haven't been able to reproduce this failure locally.</p> <p>If indeed it is due to some problem with <a href="http://review.whamcloud.com/4885" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4885</a>, for example returning from the function wait_mds_ost_sync() too quickly, shouldn't that be causing failures in all tests that use this function? This seems the only failure reported.</p> <p>Seems like the preceding test 5 must run in order to trigger the problem. I note that it did run in the failing test set. With the SLOW=no default it is skipped. I think that's why I was having difficulty reproducing the problem.</p> <p>bzzz, in comments in <a href="https://jira.whamcloud.com/browse/LU-2494" title="error: get_param: /proc/{fs,sys}/{lnet,lustre}/obdfilter/*/mds_sync: Found no match" class="issue-link" data-issue-key="LU-2494"><del>LU-2494</del></a> you said that old_sync_processed was the correct new flag to use. However I can't see it ever going to 0 when I do a sync command on the client. Seems like once it gets set to 1 near the beginning of time it stays 1 forever. If so this makes it not a good analog for the mds_sync variable in older versions. If this is true then Andreas is correct and my fix for <a href="https://jira.whamcloud.com/browse/LU-2494" title="error: get_param: /proc/{fs,sys}/{lnet,lustre}/obdfilter/*/mds_sync: Found no match" class="issue-link" data-issue-key="LU-2494"><del>LU-2494</del></a> probably led to this problem.</p> <p>yes, this represents only old requests (left from the previous boot). mds_sync (repesenting u.filter.fo_mds_ost_sync in obdfilter) can't go to 0 as well, as MDS-OST recovery happens once at the startup.</p> <p>In discussion with bzzz he suggests using the existing function wait_delete_completed_mds() to force the previous rm to finish before moving on. Apparently this kind of failure isn't new and not due to the fix for <a href="https://jira.whamcloud.com/browse/LU-2494" title="error: get_param: /proc/{fs,sys}/{lnet,lustre}/obdfilter/*/mds_sync: Found no match" class="issue-link" data-issue-key="LU-2494"><del>LU-2494</del></a>. The existing calls in test_6 of wait_mds_ost_sync and wait_destroy_complete and sync aren't sufficient to guarantee that kbytesavail is stable before starting the test. Investigating this solution. If it works in local test I will work up a patch presently.</p> <p><a href="http://review.whamcloud.com/5042" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5042</a></p> <p>another failure seen in zfs: <a href="https://maloo.whamcloud.com/test_sets/c34fa2a2-7788-11e2-987d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c34fa2a2-7788-11e2-987d-52540035b04c</a></p> <p>Lustre b2_1 client build: <a href="http://build.whamcloud.com/job/lustre-b2_1/176" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/176</a><br/> Lustre master server build: <a href="http://build.whamcloud.com/job/lustre-master/1269" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1269</a><br/> Distro/Arch: RHEL6.3/x86_64</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/93369ad0-7d78-11e2-85d0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/93369ad0-7d78-11e2-85d0-52540035b04c</a></p> <p>This is still failing in master several times every day:<br/> <a href="https://maloo.whamcloud.com/sub_tests/7b9df70e-8531-11e2-bfd3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/7b9df70e-8531-11e2-bfd3-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/3d973ca8-8532-11e2-9ab1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/3d973ca8-8532-11e2-9ab1-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/35cfc004-8513-11e2-9ab1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/35cfc004-8513-11e2-9ab1-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/7e5e9684-8512-11e2-bfd3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/7e5e9684-8512-11e2-bfd3-52540035b04c</a></p> <p>I think the failures being seen now look different. The original reported bug showed only test_6 failing, and that only after running test_5 due to SLOW=yes. New failures now show test_6 and following tests all failing. Possibly an entirely new underlying cause.</p> <p>Looks like all of the new failures are on review-zfs test runs, so it may be due to a different cause?</p> <p>Also original failure was seen regardless of fstype. New ones appear to be only on zfs, if I'm not mistaken.</p> <p>Let's task <a href="https://jira.whamcloud.com/browse/LU-2903" title="replay-ost-single test_6 and test_7 failure" class="issue-link" data-issue-key="LU-2903"><del>LU-2903</del></a> for this new failure, since it does appear that this one was only hit when test_5 was being run.</p> <p>Andreas, Looks like we are on the same page suspecting a different cause.</p> <p>Hello Oleg,</p> <p>Could you please cherry-pick the patch of <a href="http://review.whamcloud.com/5042" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5042</a> to Lustre b2_1 branch since the failure occurs in 2.1.4&lt;-&gt;2.4.0 interop testing? Thanks.</p> <p>closing again as the new issue is being tracked under <a href="https://jira.whamcloud.com/browse/LU-2903" title="replay-ost-single test_6 and test_7 failure" class="issue-link" data-issue-key="LU-2903"><del>LU-2903</del></a></p> <p>Hit this bug in interop between 1.8.9 client and 2.4 server, the server build is #1338 which should include the fix of <a href="https://jira.whamcloud.com/browse/LU-2903" title="replay-ost-single test_6 and test_7 failure" class="issue-link" data-issue-key="LU-2903"><del>LU-2903</del></a></p> <p><a href="https://maloo.whamcloud.com/test_sets/b683542a-948d-11e2-93c6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b683542a-948d-11e2-93c6-52540035b04c</a></p> <p>I think this failure is expected. The patch of <a href="http://review.whamcloud.com/5042" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5042</a> was cherry picked to b2_1, which fixed the problem for 2.1/2.4 interop. This was never done for b1_8 as far as I can see so the problem was never fixed for 1.8.9/2.4 interop.</p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b1_8/258/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/258/</a> (1.8.9-wc1)<br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)</p> <p>replay-ost-single test 6 hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/6c0c8652-15c3-11e3-87cb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6c0c8652-15c3-11e3-87cb-52540035b04c</a></p> Duplicate LU-1966 Related LU-2494 LU-2874 LU-2011 Development Rank 1|hzvfbz: Rank (Obsolete) 6130 Severity