https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7612 Lustre <p>recovery-small test_110a, test_110b, test_110c, test_110d, test_110e, and test_110f fail with </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>'lfs mkdir failed' </pre> </div></div> <p>From the test_log, we see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 110a: create remote directory: drop client req == 18:23:45 (1451154225) CMD: shadow-20vm8 lctl set_param fail_loc=0x123 fail_loc=0x123 CMD: shadow-20vm5.shadow.whamcloud.com /usr/bin/lfs mkdir -i 1 -c2 /mnt/lustre/d110a.recovery-small/remote_dir error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Bad address error: mkdir: create stripe dir '/mnt/lustre/d110a.recovery-small/remote_dir' failed CMD: shadow-20vm8 lctl set_param fail_loc=0 fail_loc=0 recovery-small test_110a: @@@@@@ FAIL: lfs mkdir failed </pre> </div></div> <p>There’s nothing interesting in the console logs on any of the nodes. </p> <p>Tests 110g, 110h, 110i and 110j fail when the other 110 tests fail and with a similar error message in the test logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Bad address error: mkdir: create stripe dir '/mnt/lustre/d110a.recovery-small/remote_dir' failed </pre> </div></div> <p>These tests has been failing since October 11, 2015. Failed test logs at:<br/> 2015-10-11 14:59:16 - <a href="https://testing.hpdd.intel.com/test_sets/c4f4aa1e-7030-11e5-b705-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c4f4aa1e-7030-11e5-b705-5254006e85c2</a><br/> 2015-10-11 23:09:20 – <a href="https://testing.hpdd.intel.com/test_sets/dfa189e2-7074-11e5-b705-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dfa189e2-7074-11e5-b705-5254006e85c2</a><br/> 2015-10-15 01:46:20 - <a href="https://testing.hpdd.intel.com/test_sets/368665d8-72e7-11e5-8fc1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/368665d8-72e7-11e5-8fc1-5254006e85c2</a><br/> 2015-10-19 02:17:13 - <a href="https://testing.hpdd.intel.com/test_sets/d8c8fb86-7610-11e5-b71e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d8c8fb86-7610-11e5-b71e-5254006e85c2</a><br/> 2015-10-29 00:59:05 - <a href="https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2</a><br/> 2015-11-05 00:18:14 - <a href="https://testing.hpdd.intel.com/test_sets/c3aa8b6e-835a-11e5-8da0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3aa8b6e-835a-11e5-8da0-5254006e85c2</a><br/> 2015-11-21 22:57:07 - <a href="https://testing.hpdd.intel.com/test_sets/4fa87c80-90ac-11e5-aaf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4fa87c80-90ac-11e5-aaf3-5254006e85c2</a><br/> 2015-11-30 19:17:14 - <a href="https://testing.hpdd.intel.com/test_sets/e9a04846-979f-11e5-b72a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e9a04846-979f-11e5-b72a-5254006e85c2</a><br/> 2015-12-23 04:36:19 - <a href="https://testing.hpdd.intel.com/test_sets/bdb1bf9e-a938-11e5-8531-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bdb1bf9e-a938-11e5-8531-5254006e85c2</a><br/> 2015-12-23 07:40:39 - <a href="https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2</a><br/> 2015-12-26 17:51:37 - <a href="https://testing.hpdd.intel.com/test_sets/b958c70a-ac01-11e5-aa1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b958c70a-ac01-11e5-aa1f-5254006e85c2</a></p> autotest review-dne-part-1 LU-7612

recovery-small tests 110a, 110b, 110c, 110d, 110e, 110f fail with 'lfs mkdir failed'

Bug Minor Open Unresolved WC Triage James Nunez dne Sun, 27 Dec 2015 21:46:30 +0000 Sun, 22 Jul 2018 09:04:43 +0000 Lustre 2.8.0 0 3 <p>There are a few other test suites for the same sessions above that fail with the LL_IOC_LMV_SETSTRIPE 'Bad address' failures:<br/> sanityn 2f, 11, 13, 14, 14a, 14c, 14d, 21, 25b, 31a, 31b, 37, 43i, 45i, 47a, 47b, 47c, 47d, 47e, 47f, 47g, 70a<br/> sanity 1, 2, 3, 5, 6g, 7a, 7b, 9, 10, 11,13, 15, 17a, 17b, 17d, 17e, 17f, 17g, 17i, 17k, 17m, 17n, 21, 22, and many others</p> <p>Upon looking at all the patches that experience this problem, most are for patch #16785 for ticket <a href="https://jira.whamcloud.com/browse/LU-2533" title="Batch RPC" class="issue-link" data-issue-key="LU-2533">LU-2533</a>.</p> <p>I just checked these failures, it seems they are all from patch <a href="http://review.whamcloud.com/#/c/16785/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16785/</a> and <a href="http://review.whamcloud.com/#/c/16969" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16969</a> (already fix the problem in the most recent patch)</p> <p>It is probably these patches problem, instead of master problem, so let's close this ticket?</p> <p>Di - I agree that most of these failures are due to patches 16785 and 16969, but there are two cases that these tests failed with this error:<br/> <a href="https://jira.whamcloud.com/browse/LU-7318" title="OUT: dynamic reply buffer" class="issue-link" data-issue-key="LU-7318"><del>LU-7318</del></a>, patch 16889 - <a href="https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2</a><br/> <a href="https://jira.whamcloud.com/browse/LU-7490" title="out_tx_write_exec()) LBUG" class="issue-link" data-issue-key="LU-7490"><del>LU-7490</del></a>, patch17199 - <a href="https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2</a></p> <p>What do you think about these failures?</p> <p><a href="https://jira.whamcloud.com/browse/LU-7318" title="OUT: dynamic reply buffer" class="issue-link" data-issue-key="LU-7318"><del>LU-7318</del></a>, patch 16889 - <a href="https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/af686e96-7dec-11e5-9c23-5254006e85c2</a><br/> Tests 110a – 110f fail with: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Not a directory</p> <p><a href="https://jira.whamcloud.com/browse/LU-7490" title="out_tx_write_exec()) LBUG" class="issue-link" data-issue-key="LU-7490"><del>LU-7490</del></a>, patch17199 <br/> <a href="https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b10e4b98-a953-11e5-b0df-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b63a926a-ae04-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b63a926a-ae04-11e5-8114-5254006e85c2</a> <br/> Tests 110a – 110j fail with: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Stale file handle<br/> <a href="https://testing.hpdd.intel.com/test_sets/f052ddd8-aec0-11e5-9134-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f052ddd8-aec0-11e5-9134-5254006e85c2</a><br/> Test 110a failed with: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Stale file handle</p> <p>And the most recent results from <a href="https://jira.whamcloud.com/browse/LU-7039" title="llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed:" class="issue-link" data-issue-key="LU-7039"><del>LU-7039</del></a>, patch 16969<br/> <a href="https://testing.hpdd.intel.com/test_sets/3994348a-aebf-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3994348a-aebf-11e5-8114-5254006e85c2</a><br/> Test 110a failed with: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Stale file handle<br/> <a href="https://testing.hpdd.intel.com/test_sets/1dd743e0-ae06-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1dd743e0-ae06-11e5-8114-5254006e85c2</a><br/> Tests 110a – 110j fail with: error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d110a.recovery-small/remote_dir' (3): Stale file handle</p> <p>It seems the newest run already pass the test, so I guess the update patch already fixed the problem. And patch 17199 is based on patch 16969. </p> <p>this issue still happens time to time, about 11 times in this year</p> <p>the latest one:<br/> <a href="https://testing.whamcloud.com/sub_tests/3c8d7b98-3d69-11e8-b45c-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/3c8d7b98-3d69-11e8-b45c-52540065bddc</a></p> Development Rank 1|hzxwuf: Rank (Obsolete) 9223372036854775807 Severity