Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-5079] conf-sanity test_47 timeout https://jira.whamcloud.com/browse/LU-5079 Lustre <p>This issue was created by maloo for Nathaniel Clark <nathaniel.l.clark@intel.com></p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/7f09a2f6-dd9d-11e3-9262-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/7f09a2f6-dd9d-11e3-9262-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/99ea9712-dc88-11e3-9450-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/99ea9712-dc88-11e3-9450-52540035b04c</a></p> <p>The sub-test test_47 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: conf-sanity 47</p> LU-5079

conf-sanity test_47 timeout

Bug Critical Resolved Fixed Jian Yu Maloo llnl patch Mon, 19 May 2014 09:43:31 +0000 Thu, 27 Nov 2014 16:56:54 +0000 Thu, 27 Nov 2014 16:56:34 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.5.4 0 16 <p>Seems that replies are being dropped after only 1s due to at_max. This is probably caused by the recently landed patch <a href="http://review.whamcloud.com/9100" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9100</a> that changes AT for the first time in a long time.</p> <p>Is the at_max to low?</p> <p>another in master:<br/> <a href="https://maloo.whamcloud.com/test_sets/e70d95f4-f2f8-11e3-b88b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e70d95f4-f2f8-11e3-b88b-52540035b04c</a></p> <p>Still failing occasionally: <a href="https://testing.hpdd.intel.com/test_sessions/a356cb0e-075d-11e4-92f3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/a356cb0e-075d-11e4-92f3-5254006e85c2</a></p> <p><a href="http://review.whamcloud.com/9100" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9100</a> introduced a regression for the case where we're sending early replies in recovery. The deadline increase (service estimate increase) is calculated as:</p> <p>Recovery case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at_measured(&svcpt->scp_at_estimate, min(at_extra, req->rq_export->exp_obd->obd_recovery_timeout / 4)); </pre> </div></div> <p>Normal case is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at_measured(&svcpt->scp_at_estimate, at_extra + cfs_time_current_sec() - req->rq_arrival_time.tv_sec); </pre> </div></div> <p>We probably want something like the following for recovery case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at_measured(&svcpt->scp_at_estimate, min(at_extra, req->rq_export->exp_obd->obd_recovery_timeout / 4) + cfs_time_current_sec() - req->rq_arrival_time.tv_sec); </pre> </div></div> <p>Alex Boyko at Xyratex pointed out the regression to me, and I believe they will submit a patch.</p> <p>patch <a href="http://review.whamcloud.com/#/c/11213/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11213/</a></p> <p>This appears to be related to the <a href="https://jira.whamcloud.com/browse/LU-5077" title="insanity test_1: out of memory on MDT in crypto_create_tfm()" class="issue-link" data-issue-key="LU-5077"><del>LU-5077</del></a> problems. Since the patch for <a href="https://jira.whamcloud.com/browse/LU-4578" title="Early replies do not honor at_max" class="issue-link" data-issue-key="LU-4578"><del>LU-4578</del></a> is in the b2_5 branch I believe this might be the source of our recovery problems. I was seeing recovery issues on 2.5 in my test bed until I applied the patch for this ticket here.</p> <p>Here is the back-ported patch for Lustre b2_5 branch: <a href="http://review.whamcloud.com/12365" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12365</a></p> <blockquote><p>patch <a href="http://review.whamcloud.com/#/c/11213/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11213/</a></p></blockquote> <p>The above patch introduced regression failure in replay-vbr test 7e.</p> <p>On master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/92fe44e0-5b80-11e4-a35f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/92fe44e0-5b80-11e4-a35f-5254006e85c2</a></p> <p>On b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4d02235e-59c2-11e4-aa32-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4d02235e-59c2-11e4-aa32-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/25213bfe-59c2-11e4-aa32-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/25213bfe-59c2-11e4-aa32-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/3238224a-59bc-11e4-816e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3238224a-59bc-11e4-816e-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/166cc778-59bc-11e4-816e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/166cc778-59bc-11e4-816e-5254006e85c2</a></p> <p>I tried <a href="http://review.whamcloud.com/12365" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12365</a> on top of LLNL's 2.5.3-1chaos tag, and saw lots of problems. I don't know if they were actually cause by it or not. Experience is recorded in <a href="https://jira.whamcloud.com/browse/LU-5805" title="tgt_recov blocked and "waking for gap in transno"" class="issue-link" data-issue-key="LU-5805"><del>LU-5805</del></a>.</p> <blockquote><p>patch <a href="http://review.whamcloud.com/#/c/11213/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11213/</a></p></blockquote> <p>I tried to manually run replay-vbr test and found it finally passed and took about 8 hours:<br/> <a href="https://testing.hpdd.intel.com/test_sets/18fb6cc0-5eb1-11e4-a2a3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/18fb6cc0-5eb1-11e4-a2a3-5254006e85c2</a></p> <p>Among the sub-tests, test 7e took 3876s, which exceeded the 3600s timeout value set by autotest system. That was why the test was stopped in autotest runs.</p> <p>Patch landed to Master.</p> <p>Here is the patch for master branch to speed up replay-vbr test 7*: <a href="http://review.whamcloud.com/12490" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12490</a><br/> And here is the test result: <a href="https://testing.hpdd.intel.com/test_sets/0ae4ab72-5fcb-11e4-895a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0ae4ab72-5fcb-11e4-895a-5254006e85c2</a></p> <p>With the above patch, total run time for replay-vbr test 7* was reduced from <b>18796s</b> to <b>4742s</b>.</p> <p>Reopen this bug until the replay-vbr test patch has been landed to master, otherwise it will not be tracked properly.</p> <p>It has landed to master now</p> <p>The patches caused regression failures <a href="https://jira.whamcloud.com/browse/LU-5900" title="replay-dual test_11: rm: cannot remove `/mnt/lustre/f11.replay-dual-[1-5]': No such file or directory" class="issue-link" data-issue-key="LU-5900"><del>LU-5900</del></a>, <a href="https://jira.whamcloud.com/browse/LU-5901" title="replay-dual test_15a: import is not in FULL state" class="issue-link" data-issue-key="LU-5901"><del>LU-5901</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5902" title="replay-dual test_20: FAIL: recovery time is growing 215 > 107" class="issue-link" data-issue-key="LU-5902"><del>LU-5902</del></a> on master and b2_5 branches.</p> <p>Alexander, could you please take a look at <a href="https://jira.whamcloud.com/browse/LU-5900" title="replay-dual test_11: rm: cannot remove `/mnt/lustre/f11.replay-dual-[1-5]': No such file or directory" class="issue-link" data-issue-key="LU-5900"><del>LU-5900</del></a>, <a href="https://jira.whamcloud.com/browse/LU-5901" title="replay-dual test_15a: import is not in FULL state" class="issue-link" data-issue-key="LU-5901"><del>LU-5901</del></a>, <a href="https://jira.whamcloud.com/browse/LU-5902" title="replay-dual test_20: FAIL: recovery time is growing 215 > 107" class="issue-link" data-issue-key="LU-5902"><del>LU-5902</del></a> failures? This looks to be caused by the <a href="http://review.whamcloud.com/11213" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11213</a> patch backported to b2_5 (<a href="http://review.whamcloud.com/12365" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12365</a>). Is this something that is just causing the tests to fail that won't affect real users, or is there something bad in b2_5 that will cause recovery problems for users also?</p> <p>I just pushed a patch to Lustre b2_5 branch to fix the calculation of of service_time in max_recovery_time(): <a href="http://review.whamcloud.com/12714" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12714</a>.<br/> I'll check the test results.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12714" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12714</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> tests: fix service_time in max_recovery_time()<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 3<br/> Commit: 264ea0643d46240b8b45dd673aeff3b4fe76bd10</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12724" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12724</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> tests: fix service_time in max_recovery_time()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 2<br/> Commit: 4c0c53265dc9fb17bb8999548d656103ced58928</p> <p>Test results showed that by increasing the time in max_recovery_time() in test framework, the failures in <a href="https://jira.whamcloud.com/browse/LU-5900" title="replay-dual test_11: rm: cannot remove `/mnt/lustre/f11.replay-dual-[1-5]': No such file or directory" class="issue-link" data-issue-key="LU-5900"><del>LU-5900</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5901" title="replay-dual test_15a: import is not in FULL state" class="issue-link" data-issue-key="LU-5901"><del>LU-5901</del></a> were resolved. And by adding some margin into the recovery time comparison condition, the failure in <a href="https://jira.whamcloud.com/browse/LU-5902" title="replay-dual test_20: FAIL: recovery time is growing 215 > 107" class="issue-link" data-issue-key="LU-5902"><del>LU-5902</del></a> was resolved.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12724" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12724</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> tests: fix service_time in max_recovery_time()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 3<br/> Commit: 1c58e970d684edfefba3725f869ffd36c6a4475e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12724/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12724/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> tests: fix service_time in max_recovery_time()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9bb24bf1ce4977b32d4bf9b55cef5a25072cef5e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12714/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12714/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5079" title="conf-sanity test_47 timeout" class="issue-link" data-issue-key="LU-5079"><del>LU-5079</del></a> tests: fix service_time in max_recovery_time()<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 4e8def3e32ad76808a5b8336d43430a5318e20aa</p> Duplicate LU-5773 Related LU-5077 LU-5805 LU-4578 LU-5358 LU-5077 LU-5900 LU-5901 LU-5902 LU-5803 LU-5724 Development Rank 1|hzwmpj: Rank (Obsolete) 14015 Severity