https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4002 Lustre <p>In 23c197908902183d5f88d3f431da6cde9c290e07 <a href="https://jira.whamcloud.com/browse/LU-3811" title="non-root users cannot archive files, root cannot archive non-root users&#39; files" class="issue-link" data-issue-key="LU-3811"><del>LU-3811</del></a> hsm: handle file ownership and timestamps, I added a stat() of the file being restored to the CT's restore path. This is to ensure that the volatile file is given the correct ownership and timestamps before the restore, and is required for the layout swap to succeed. However this introduces a potential for deadlock vs unlink() and other operations. Consider the following sequence of operations on a single file:</p> <ol> <li>Client sends restore, CDT takes and holds EX LAYOUT lock.</li> <li>Client sends unlink, handler sleeps on EX FULL lock.</li> <li>CDT sends restore action to CT.</li> <li>CT begins restore, sends getattr (from stat()), handler sleeps on PR LOOKUP,UPDATE,PERM lock.</li> </ol> <p>We have a similar deadlock with rename-onto.</p> <p>I think the simplest way out of this mess would be to lock fewer bits in the unlink handler. Can anyone say why unlink should invalidate cached layout? An open unlinked file is still valid for IO.</p> LU-4002

HSM restore vs unlink deadlock

Technical task LU-3647 Blocker Resolved Fixed John Hammond John Hammond HSM Tue, 24 Sep 2013 18:41:24 +0000 Tue, 25 Jan 2022 20:56:30 +0000 Wed, 9 Oct 2013 08:22:39 +0000 Lustre 2.5.0 Lustre 2.5.0 0 6 <p>I wouldagree that unlink does not need to invalidate the layout</p> <p>Is it possible for unlink to grab LOOKUP only? The only side effect I can think of now is that there are some caching locks on the client side won't be revoked. But this can be easily fixed.</p> <p>In general it won't be enough just to remove LAYOUT from unlink.Assume restore takes EX LAYOUT, some other operation tries to take PR LAYOUT | LOOKUP | ... and waits. Then unlink tries to take EX LOOKUP | .... Then stat (from the CT) tries to take LOOKUP | UPDATE | ... and deadlocks.</p> <p>As I keep saying, any operation that requires two lock is dangerous.</p> <p>Note that unlink (and rename onto) should take more than just LOOKUP, since it modifies link count and timestamps.</p> <p>Could we do something sane seeming like having the MDT send the attributes that the CT is getting from stat?</p> <p>Please see <a href="http://review.whamcloud.com/7792" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7792</a>.</p> <p>I suggest that this be considered a blocker for 2.5.0. It is easy to imagine situations where users will trigger this deadlock. Faced with a long running restore on a file (which to the user may just seem like an unresponsive console or FS) the user may logout, login, and unlink the released file (since perhaps it is easy to regenerate anyway).</p> <p>Patch landed to Master. If more work is needed in this ticket, please let me know and I will reopen this ticket.</p> <p>Per recent comments in <a href="https://jira.whamcloud.com/browse/LU-4053" title="client leaking objects/locks during IO" class="issue-link" data-issue-key="LU-4053"><del>LU-4053</del></a>, this is causing a lot of layout locks to be left on the client after an inode is unlinked. Ideally, the layout lock would be revoked if this is the last reference to the inode (last link and file is not opened). The main question is whether it is possible to know this in advance? Unlike the open-unlinked handling, at worst this will revoke an extra lock if there is a race and another thread opens the file just before it is unlinked, so I think it is better to handle the common case more efficiently.</p> <p>Is there any way to know in advance if HSM is processing this file and not try to revoke the layout lock in this case?</p> <p>Might have jumped the gun on this. Closing it again until we know it is the culprit. </p> <p>Hi Andreas, this is a known issue but we still decided to land the patch because the deadlock issue is more severe.</p> <p>The problem is that when the LOOKUP lock is revoked, we don't know if this is because the file is being unlinked or renamed. However, leaving some locks in cache may not be a problem because if the system is active, the locks will be discarded by LRU soon or later.</p> <p>I believe that we could revert this patch if the HSM coordinator would send the ownership and timestamps to use on the volatile file along with the restore action. Then the copytool will not need to stat() the original file. The restoring layout swap will check that the ownerships agree protecting us from a TOCTTOU issue.</p> <p>Please see <a href="http://review.whamcloud.com/7927" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7927</a> for a sketch of this approach.</p> <p>Note to self. After <a href="http://review.whamcloud.com/13750" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13750</a> (<a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a> hsm: use IOC_MDC_GETFILEINFO in restore) the copytool uses IOC_MDC_GETFILEINFO/MDS_GETATTR_NAME to get the attributes of the released file which bypasses LLDM and does not have the issue described above.</p> Related LU-4053 LU-4727 Development Rank 1|hzw3tr: Rank (Obsolete) 10714