https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6203 Lustre <p>sanity-hsm test 251 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-26vm10 pkill -INT -x lhsmtool_posix CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 Copytool still running on shadow-26vm10 CMD: shadow-26vm10 pgrep -x lhsmtool_posix shadow-26vm10: 7902 sanity-hsm test_251: @@@@@@ FAIL: Copytool failed to stop in 20s ... </pre> </div></div> <p>Maloo reports:<br/> <a href="https://testing.hpdd.intel.com/test_sets/49281842-a9ef-11e4-8c6f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/49281842-a9ef-11e4-8c6f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/4cac5380-aa11-11e4-a5c6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4cac5380-aa11-11e4-a5c6-5254006e85c2</a></p> Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/112/">https://build.hpdd.intel.com/job/lustre-b2_5/112/</a><br/> FSTYPE=zfs LU-6203

sanity-hsm test 251: FAIL: Copytool failed to stop in 20s

Bug Blocker Resolved Fixed Bruno Faccini Jian Yu 22pl zfs Tue, 3 Feb 2015 07:51:05 +0000 Mon, 14 Sep 2015 16:01:25 +0000 Wed, 11 Mar 2015 12:42:57 +0000 Lustre 2.5.4 Lustre 2.8.0 0 10 <p>This is a regression failure introduced by the following commit in Lustre b2_5 build #112:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Commit 97fc8c8caf41e9d74cdb1e373f19c907ed8481b2 by Oleg Drokin LU-5622 tests: check/wait for copytool death Seems that copytool death/kill may take more time so this condition must be handled in sanity-hsm copytool_cleanup() function to avoid situations where copytool will then not be restarted, but only signaled, in next copytool_setup(). This patch is back-ported from the following one: Lustre-commit: 6facf3953b170832200ca9c111398da8feecd281 Lustre-change: http://review.whamcloud.com/11922 Signed-off-by: Bruno Faccini &lt;bruno.faccini@intel.com&gt; Change-Id: Ia817936eb030386dbe539ec8d5297812f4b6fff2 Reviewed-on: http://review.whamcloud.com/12967 Tested-by: Jenkins Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt; Reviewed-by: James Nunez &lt;james.a.nunez@intel.com&gt; Reviewed-by: Henri Doreau &lt;henri.doreau@cea.fr&gt; Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt; </pre> </div></div> <p>Hi Bruno,<br/> Could you please take a look at the failure? Thank you.</p> <p>The zfs full group test session was not run on master branch, so we do not know whether the failure exists on master branch or not for now.</p> <p>Yu Jian, thanks for all this research work already!!<br/> Will try to reproduce with a ZFS-only configuration and also have a look to the logs of the different cases you pointed to.</p> <p>Bruno, Yu Jian, is this also happening on master, or only on b2_5?</p> <p>Andreas: no not on master. Based on Maloo reports search, latest test_251 sub-test failures in master have occurred about a year ago and at this time, my patch for <a href="https://jira.whamcloud.com/browse/LU-5622" title="copytool_cleanup function should check/wait for copytool death" class="issue-link" data-issue-key="LU-5622"><del>LU-5622</del></a> was far to be integrated!, and they were linked to <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a></p> <p>There has only been a bunch of 8 failures, between 2014-12-30 07:31:46 UTC and 2015-01-30 14:05:49 UTC and after my patch for <a href="https://jira.whamcloud.com/browse/LU-5622" title="copytool_cleanup function should check/wait for copytool death" class="issue-link" data-issue-key="LU-5622"><del>LU-5622</del></a> has been integrated, all with b2_5 (2 occurrences) or b_ieel2_0 (6 occurrences) branches, and only when using zfs targets. But there are also frequent success for b2_5/b_ieel2_0 branches using zfs!</p> <p>I have still not been able to reproduce the problem running with b2_5 build #112, that has been reported to trigger the problem.<br/> I have also analyzed the logs of the different failures and it appears that :<br/> _ the copytool's PID still being reported as alive is either the one running the archive action or the main one.<br/> _ each time the copytool log shows that at the time of the kill, the archive action seems to have a slow start and the last log line is "processing file ...", when the "archiving ..."/"saving stripe info of ..."/"start copy of ..." log lines are present in the successful run logs.<br/> _ the Agent debug log shows that the PID running the archive action has been stuck awaiting for an OST_GETATTR request to be replied, during a variable period of time but each time exceeding the 20s allowed to wait for copytool death.<br/> _ during that time the concerned OSS has been trying/waiting to cancel a contending lock from the Client that has created the file being archived.</p> <p>So could this be the consequence of some ZFS/Network config/performance related causing the file's dirty page flush, occuring at lock cancel time, to take more than 20s under some circumstances ?<br/> I think that a fix for these failures could be either to raise the timer waiting for copytool death (40s?) or ensure dirty data/blocks, during file creation, have been flushed before to start the archive operation (with "cancel_lru_locks osc"?).</p> <p>Also, I wonder if the priority of this ticket should be kept as Blocker ??</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13646" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13646</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6203" title="sanity-hsm test 251: FAIL: Copytool failed to stop in 20s" class="issue-link" data-issue-key="LU-6203"><del>LU-6203</del></a> tests: early lock cancel to allow early copytool death<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cc1f9bc9c062e07515fe6c08e358a703cee116ae</p> <p><a href="http://review.whamcloud.com/13646" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13646</a> implements early lock cancel solution to speed-up copytool death.</p> <p>The zfs full group test session was not run on master branch. Maybe this is the reason that we did not search out failure instances on master branch on Maloo.</p> <p>In the following report on b2_5 branch, many sub-tests failed with this issue:<br/> <a href="https://testing.hpdd.intel.com/test_sets/97fd06d8-ac1c-11e4-992b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/97fd06d8-ac1c-11e4-992b-5254006e85c2</a></p> <p>The failure started affecting patch review testing on Lustre b2_5 patches:<br/> <a href="https://testing.hpdd.intel.com/test_sets/66e27944-acde-11e4-872a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/66e27944-acde-11e4-872a-5254006e85c2</a></p> <p>Concerning the "https://testing.hpdd.intel.com/test_sets/97fd06d8-ac1c-11e4-992b-5254006e85c2" case, the 'Copytool failed to stop in 20s ...' errors/symptoms there looks more as an additional consequence (due to cleanup()/copytool_cleanup() execute as a trap, set in copytool_cleanup(), upon exit) of previous problem within same sub-test or even preceeding sub-tests :</p> <p> _ 1st sub-test reported to have failed for 'Copytool failed to stop in 20s ...' is test_33, but it has failed previously with "sanity-hsm test_33: @@@@@@ FAIL: request on 0x200007931:0x2b:0x0 is not SUCCEED on mds1" waiting for archive to complete/succeed ...</p> <p> _ 2nd sub-test reported to have failed for 'Copytool failed to stop in 20s ...' is test_60, but it has failed previously with "sanity-hsm test_60: @@@@@@ FAIL: Timed out waiting for progress update!" waiting for a progress update during archive.</p> <p> _ 3rd sub-test reported to have failed for 'Copytool failed to stop in 20s ...' is test_70, but it is the next just after test_60 and since its 1st cmd is a copytool_cleanup that is likely to have encountered the same problem than the preceeding.</p> <p> _ 4th sub-test reported to have failed for 'Copytool failed to stop in 20s ...' is test_71, but it is the next just after test_70 and since its 1st cmd is a copytool_cleanup that is likely to have encountered the same problem than the 2 preceeding.</p> <p> _ 5th sub-test reported to have failed for 'Copytool failed to stop in 20s ...' is test_103, and according to its specific logs it is the only one that seems to have triggered the same scenario (huge delay during lock flush/cancel processing) I have already described in my previous update. So it may be an other potential subject to the same change than for test_251.</p> <p>After more Lustre debug log reading, it seems that the "huge delay during lock flush/cancel processing" that seems to be the root cause of the problem is mainly on the OSS side, after the Client has handled the Blocking callback and sent back its Cancel of lock to the OSS. The thread handling it on the OSS can then spend multiple tens of seconds in ldlm_request_cancel()<del>&gt;ldlm_lock_cancel()</del>&gt;ldlm_cancel_callback()-&gt; .... and <br/> highly probably tgt_blocking_ast()<del>&gt;tgt_sync()</del>&gt;dt_object_sync()-&gt;osd_object_sync(). So is this finally some kind of ZFS performance issue?</p> <p>Another failure instance on b2_5:<br/> <a href="https://testing.hpdd.intel.com/test_sets/465bb228-bbfa-11e4-a79b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/465bb228-bbfa-11e4-a79b-5254006e85c2</a></p> <p>Last 2 failures occurrences again show the same delay of copytool death/cleanup due to being stuck, during tens of seconds, because waiting for lock flush/cancel (between Client, who have created file and asked for its archive, and OST<span class="error">&#91;s&#93;</span>) upon archiving file.</p> <p>This seem to only occur on ZFS until now, and I have verified that my patch, adding a cancel_lru_locks() before hsm_archive allows the copytool copy to start immediatelly, which is normally delayed to allow for previous Client's lock flush/cancel.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13646/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13646/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6203" title="sanity-hsm test 251: FAIL: Copytool failed to stop in 20s" class="issue-link" data-issue-key="LU-6203"><del>LU-6203</del></a> tests: early lock cancel to allow early copytool death<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 446ad1ef8be5b282224336817936043b42a10fee</p> <p>Landed for 2.8</p> Related LU-5622 LU-7150 Development Rank 1|hzx5hz: Rank (Obsolete) 17337 Severity