https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4419 Lustre <p>This issue was created by maloo for nasf &lt;fan.yong@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/e0c22550-6db7-11e3-a191-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/e0c22550-6db7-11e3-a191-52540035b04c</a>.</p> <p>Slave MDT cannot allocate super-sequence:</p> <p>07:41:38:Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req == 07:40:37 (1387986037)<br/> 07:41:38:LustreError: 31316:0:(fid_handler.c:285:__seq_server_alloc_meta()) srv-lustre-MDT0001: Can't allocate super-sequence, rc -5<br/> 07:41:38:Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_110a: @@@@@@ FAIL: lfs mkdir failed <br/> 07:41:38:Lustre: DEBUG MARKER: recovery-small test_110a: @@@@@@ FAIL: lfs mkdir failed</p> LU-4419

Test failure on test suite recovery-small, subtest test_110a

Bug Minor Closed Duplicate Mikhail Pershin Maloo Sun, 29 Dec 2013 06:51:21 +0000 Sun, 22 Jul 2018 09:01:06 +0000 Sun, 22 Jul 2018 09:01:06 +0000 0 7 <p>We have hit the failure several times:</p> <p><a href="https://maloo.whamcloud.com/test_sets/e0c22550-6db7-11e3-a191-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e0c22550-6db7-11e3-a191-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/e553b5f8-6f2d-11e3-ad93-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e553b5f8-6f2d-11e3-ad93-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/fe06d80c-6ebd-11e3-ad93-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fe06d80c-6ebd-11e3-ad93-52540035b04c</a></p> <p>Hmm, I am not sure this is a new bug or only exists in your or Mike's patch series, since I do not see it exists in other's patch. Please correct me, if I am wrong. Are your patches still dependent?</p> <p>I cannot search the test results history because of Maloo issues. The first known failure instance was found in Mike's patch. But I do not think it is special issue in such patch, but more like general master bug. Because his patch does not touch MDT/FID stack. Current, LFSCK patches still depends on Mike's patch.</p> <p>oh, it is not about MDT/FID stack. The failure is because the connection is somehow broken between other MDTs/OSTs to MDT0, which cause these target can not allocate the new FID sequence from MDT0. Hmm if your patch still depends on Mike's patch, it is probably Mike's patch problem, since I never saw this problem in current master and even in the run of other people's patch.</p> <p>another instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/1e071d22-706e-11e3-9fe0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1e071d22-706e-11e3-9fe0-52540035b04c</a></p> <p>probably I've found the source of problem, let's wait for the latest patch test results, <a href="http://review.whamcloud.com/#/c/7383/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7383/</a></p> <p><a href="https://maloo.whamcloud.com/test_sessions/ca3dea64-7751-11e3-943d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/ca3dea64-7751-11e3-943d-52540035b04c</a></p> <p>Now it works as expected. The problem was the lost chunk of code with OBD_FAIL_CHECK needed for tests.</p> <p>Another instance <a href="https://testing.hpdd.intel.com/test_sets/2e799af4-1942-11e4-8c4a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2e799af4-1942-11e4-8c4a-5254006e85c2</a>.</p> <p>22:16:47:Lustre: DEBUG MARKER: == recovery-small test 110a: create remote directory: drop client req == 22:14:17 (1406870057)<br/> 22:16:47:Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1<br/> 22:16:47:Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1<br/> 22:16:47:Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1<br/> 22:16:47:LustreError: 167-0: lustre-MDT0000-lwp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.<br/> 22:16:47:LustreError: Skipped 2 previous similar messages<br/> 22:16:47:LustreError: 16796:0:(fid_handler.c:284:__seq_server_alloc_meta()) srv-lustre-MDT0001: Can't allocate super-sequence, rc -5<br/> 22:16:47:Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_110a: @@@@@@ FAIL: lfs mkdir failed <br/> 22:16:47:Lustre: DEBUG MARKER: recovery-small test_110a: @@@@@@ FAIL: lfs mkdir failed<br/> 22:16:47:Lustre: DEBUG MARKER: /usr/sbin/lctl dk &gt; /logdir/test_logs/2014-07-31/lustre-reviews-el6-x86_64-<del>review-dne-part-1</del>-2_7_1_<em>25543</em>_-70280407083100-154359/recovery-small.test_110a.debug_log.$(hostname -s).1406870060.log;<br/> 22:16:47: dmesg &gt; /logdir/test_logs/2014-07-31/lustre-revi<br/> 22:16:47:Lustre: DEBUG MARKER: /usr/sbin/lctl mark == recovery-small test 110b: create remote directory: drop Master rep == 22:14:24 &#40;1406870064&#41;</p> <p>Another instance on 2.7.0-RC2. Logs at <a href="https://testing.hpdd.intel.com/test_sets/20cb4134-bf80-11e4-881f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/20cb4134-bf80-11e4-881f-5254006e85c2</a></p> <p>Client test log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 110a: create remote directory: drop client req == 08:02:47 (1425139367) CMD: onyx-41vm3 lctl set_param fail_loc=0x123 fail_loc=0x123 CMD: onyx-41vm6.onyx.hpdd.intel.com /usr/bin/lfs mkdir -i 1 -c2 /mnt/lustre/d110a.recovery-small/remote_dir error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Input/output error error: mkdir: create stripe dir '/mnt/lustre/d110a.recovery-small/remote_dir' failed CMD: onyx-41vm3 lctl set_param fail_loc=0 </pre> </div></div> <p>MDT same as the log John posted above.</p> <p>This test is still failing occasionally. Recent failures are:<br/> 2015-04-27 23:46:54 - <a href="https://testing.hpdd.intel.com/sub_tests/3da2d940-ed41-11e4-b3fc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/3da2d940-ed41-11e4-b3fc-5254006e85c2</a><br/> 2015-05-06 15:04:26 - <a href="https://testing.hpdd.intel.com/sub_tests/73c503ac-f40a-11e4-b108-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/73c503ac-f40a-11e4-b108-5254006e85c2</a><br/> 2015-05-14 09:21:11 - <a href="https://testing.hpdd.intel.com/sub_tests/77b99d6e-fa23-11e4-8c8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/77b99d6e-fa23-11e4-8c8b-5254006e85c2</a><br/> 2015-05-17 00:07:52 - <a href="https://testing.hpdd.intel.com/sub_tests/fcaf8172-fc31-11e4-a658-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/fcaf8172-fc31-11e4-a658-5254006e85c2</a><br/> 2015-05-26 10:10:02 - <a href="https://testing.hpdd.intel.com/sub_tests/283e0a20-0399-11e5-a102-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/283e0a20-0399-11e5-a102-5254006e85c2</a></p> <p>Mike, <br/> Would you please look at these recent test failures?<br/> Thank you.</p> <p>the releated test issue was fixed and remaining problems are <a href="https://jira.whamcloud.com/browse/LU-7612" title="recovery-small tests 110a, 110b, 110c, 110d, 110e, 110f fail with &#39;lfs mkdir failed&#39; " class="issue-link" data-issue-key="LU-7612">LU-7612</a></p> Development Rank 1|hzwbxr: Rank (Obsolete) 12134 Severity