https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3834 Lustre <p>In the restore case of hsm_cdt_request_completed(), if the copytool returned success but the layout swap fails then we get an unreadable file with HS_RELEASED clear but LOV_PATTERN_F_RELEASED set.</p> <p>Perhaps the new HSM attributes should be applied to the volatile object before layout swap, and hsm_swap_layouts() should call mo_swap_layouts() with SWAP_LAYOUTS_MDS_HSM set.</p> LU-3834

hsm_cdt_request_completed() may clear HS_RELEASED on failed restore

Technical task LU-3647 Critical Resolved Fixed Bruno Faccini John Hammond HSM Mon, 26 Aug 2013 19:48:32 +0000 Wed, 24 Feb 2016 17:25:14 +0000 Mon, 20 Jan 2014 16:52:36 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.1 0 6 <p>John,<br/> Sorry, but HSM code is still new for me, so I need some clarifications here ...</p> <p>In this particular case of error handling, there is no other option than to return the file to its released state, is'nt it ? <br/> So is this what you mean by having "hsm_swap_layouts() call mo_swap_layouts() with SWAP_LAYOUTS_MDS_HSM set", as during release ?</p> <p>John,<br/> Can you also detail me the scenario/conditions when you encountered such problem already ? As we discussed I may then be able to reproduce it or do some kind of error injection.</p> <p>Bruno,</p> <p>(I am just restating what I said on the call today.)</p> <p>In mdt_hsm_release() we call mdd_swap_layouts() with the SWAP_LAYOUTS_MDS_HSM flag, which causes the HSM xattrs to be handled along with the LOV xattrs, and rolls back both xattrs on failure.</p> <p>Contrast this with hsm_cdt_request_completed() which calls mdt_hsm_attr_set() and hsm_swap_layouts(). hsm_swap_layouts() then calls mdd_swap_layouts() with no flags. In this case if the layout swap fails then we do not restore the HSM xattr to its previous (released set) state.</p> <p>I have not checked but I suspect that may be possible to remodel the handing of restore after that of release and thereby avoid this inconsistency when layout swap fails.</p> <p>If this is a simple fix, then we can work out a patch for this. Otherwise I'd like to put the resource on something else because it's unlikely for swap_layout to fail anyway.</p> <p>Ok, I see where this can be fixed now, thank's John. But now, and to save time before submit patch, what is the preferred way to do this ? :</p> <p> _ always call mo_swap_layouts() with SWAP_LAYOUTS_MDS_HSM flag from hsm_swap_layouts(), since (actually?) hsm_swap_layouts() is only called from mdt_hsm_update_request_state() for a RESTORE op.</p> <p> _ add a new flags to hsm_swap_layouts() to enable SWAP_LAYOUTS_MDS_HSM flag use or not during call to mo_swap_layouts().</p> <p>&gt; _ always call mo_swap_layouts() with SWAP_LAYOUTS_MDS_HSM flag from hsm_swap_layouts(), since (actually?) hsm_swap_layouts() is only called from mdt_hsm_update_request_state() for a RESTORE op.</p> <p>This seems better to me.</p> <p>1st patch attempt is at <a href="http://review.whamcloud.com/7631" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7631</a>. <br/> Unfortunately it failed in multiple auto-tests and needs at least to re-base. <br/> But also failed in several sub-tests of sanity-hsm, which is highly suspect ... <br/> Currently under investigations.</p> <p>Latest patch-set #6 auto-tests failures appear not related to this patch but to multiple issues already addressed in others tickets (<a href="https://jira.whamcloud.com/browse/LU-4093" title="sanity-hsm test_24d: wanted &#39;SUCCEED&#39; got &#39;WAITING&#39;" class="issue-link" data-issue-key="LU-4093"><del>LU-4093</del></a>, <a href="https://jira.whamcloud.com/browse/LU-4086" title="Test failure on test suite sanity-hsm, subtest test_33" class="issue-link" data-issue-key="LU-4086"><del>LU-4086</del></a>, …). Some of them have heir patches that landed until now, so I will rebase it again …</p> <p>I found a possible bug in my original patch version causing layout-lock not to be released when restore is canceled … Just submitted patch-set #8 to fix this, will see if is passes auto-tests (particularly sanity-hsm/test_33 which was timing-out due to md5sum process never ending!!…).</p> <p>I am wondering if I should also add some error injection to simulate SWAP_LAYOUT failure during restore ??</p> <p>I will also push a new patch-set #8 to address John's last comment and convert to usual error handling style.</p> <p>Some update, after I added fault-injection (force -ENOENT in the middle of mdd_swap_layouts() to cause layouts swap back) and associated sub-test test_12o within patch-set #8.</p> <p>test_12o fails due to "diff" command, that caused the implicit restore, to be successful when it is expected to fail because of the fault-injection. Strange is that the Restore operation has been marked as failed, the Copytool received the error, and file still has the "released" flag set!!</p> <p>I wonder if there could be some issue in mdd_swap_layouts() causing this unexpected behavior ?</p> <p>Hehe, finally I found that my fault-injection code itself introduced some problem because being added after the volatile/2nd file layout change and not reverting it to mimic the error !! This caused the restored datas to be available as if restore succeed …</p> <p>I changed this in patch-set #13, and now new sub-test test_12o runs fine, returning errors on both copytool (ENOTSUPP, injected!) and client (ENODATA) sides with layout-swap fault-injection, and next restore attempt without fault-injection to be successful.</p> <p>Will run with build+patch locally and see if I can still reproduce the Volatile object leak on MDT, seen as part of this ticket and <a href="https://jira.whamcloud.com/browse/LU-4293" title="lfs_migrate is failing with a volatile file Operation not permitted error" class="issue-link" data-issue-key="LU-4293"><del>LU-4293</del></a>.</p> <p>patch <a href="http://review.whamcloud.com/7631" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7631</a> has landed. Closing.</p> <p>Patch was only landed to master and not b2_5. In the future, this type of patch should be cherry-picked to b2_5 so that it is fixed in the maintenance release.</p> <p>Hello Andreas,<br/> I am sorry if I missed to do something here, to be honest actually I mainly focus to get the patch done for the branch where problem has been reported. But then should I create a new patch version for each affected version listed?</p> <p>Bruno, the patch was marked as affecting the 2.5.0 release. I'm just going through patches that have landed to master and trying to see which ones need to be landed for 2.5.1 that have not been landed there, since that is the long-term maintenance release. If you are closing a but then you should consider if it is fixing a problem that is serious and may affect earlier versions of Lustre and should land on the maintenance release. In many cases, Oleg can cherry-pick the patch directly to b2_5 without putting it through Gerrit/Jenkins/autotest again, but he needs to know to do this.</p> <p>Ok thanks Andreas, I understand now that I need to take care of this because it is also under my responsibility, if a patch is required for earlier versions, to either create+push a new patch for each other versions or ask Oleg to cherry-pick the original patch for each other versions.</p> <p>I don't know why but I thought that the patch integration/release decision was done by other people (you, Oleg, Peter, …), this may simply be you are doing this verification work very requently and do the job for lazy guys like me!!</p> <p>Andreas, you b2_5 patch for this ticket at <a href="http://review.whamcloud.com/9212" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9212</a>, has found a flaw in sanity-hsm/test_12o (from original patch <a href="http://review.whamcloud.com/7631" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7631</a> from this ticket too !!) during auto-tests session.</p> <p>This new problem is tracked within <a href="https://jira.whamcloud.com/browse/LU-4613" title="Failure on test suite sanity-hsm test_12o: request on 0x200000bd1:0xf:0x0 is not SUCCEED on mds1" class="issue-link" data-issue-key="LU-4613"><del>LU-4613</del></a> where I already pushed a patch to master (<a href="http://review.whamcloud.com/9235" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9235</a>), since #7631 has already landed to master, but what should we do for the b2_5 version you just pushed ?</p> Development Rank 1|hzvz87: Rank (Obsolete) 9919