https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3671 Lustre <p><tt>mdd_acl_set()</tt> sets the ACL xattr synchronously if <tt>/proc/fs/lustre/mdd/lustre-MDT0000/sync_permission</tt> is set (it is set by default). This can have a large performance impact for operations like <tt>cp -a</tt> or <tt>tar --xattr</tt>. Should this be the default behavior? How does this relate to other permissions-related changed like chown, chmod, etc.? It would be good to explain the security implications of this setting in code comments and in the Lustre manual.</p> <p>Some background information:</p> <p><a href="http://wiki.lustre.org/index.php/Architecture_-_Version_Based_Recovery#Permissions" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/index.php/Architecture_-_Version_Based_Recovery#Permissions</a><br/> <a href="https://projectlava.xyratex.com/show_bug.cgi?id=15390" class="external-link" target="_blank" rel="nofollow noopener">https://projectlava.xyratex.com/show_bug.cgi?id=15390</a><br/> <a href="http://git.whamcloud.com/gitweb?p=lustre.git;a=commit;h=0509770778396825ba1d7f1b42be5993aac1d6f3" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/gitweb?p=lustre.git;a=commit;h=0509770778396825ba1d7f1b42be5993aac1d6f3</a></p> <p>LLNL-bug-ID: TOSS-2207</p> LU-3671

why are permission changes synchronous?

Bug Critical Resolved Fixed Lai Siyao Ned Bass zfs Tue, 30 Jul 2013 18:39:57 +0000 Tue, 6 Nov 2018 10:53:57 +0000 Wed, 4 Sep 2013 18:56:01 +0000 Lustre 2.4.0 Lustre 2.5.0 0 11 <p>This (and/or <a href="https://jira.whamcloud.com/browse/LU-3660" title="Can&#39;t disable ACL support with ZFS MDT" class="issue-link" data-issue-key="LU-3660"><del>LU-3660</del></a>) is LLNL's top support priority.</p> <p>This setting does apply to chmod, chown, etc. In a VM environment with ZFS backend, chmod or chown of 60 files takes about 15s with sync_permission=1, versus 100ms with sync_permission=0.</p> <p>Lai</p> <p>This seems related to <a href="https://jira.whamcloud.com/browse/LU-3660" title="Can&#39;t disable ACL support with ZFS MDT" class="issue-link" data-issue-key="LU-3660"><del>LU-3660</del></a> which you are already working on</p> <p>Peter</p> <p>To clarify our immediate priority here, how risky would it be to turn off <tt>sync_permission</tt>? We're feeling significant pain due to this issue on our production filesystems, so we'd like to take prompt action if its reasonably safe to do so.</p> <p>My impression is that we're trying to prevent something like an <tt>open()</tt> getting incorrectly replayed before a <tt>chmod()</tt> during recovery, allowing a client to circumvent permissions. This seems fairly low risk, but I'm not sure I understand this issue completely.</p> <p>IIRC, there is no harm or bug related to sync_permission, that is just an attempt to make any permission changes persistent and be not lost due to recovery. The sync_permission guarantees that changes are persistently stored when reply is got. With sync_permission == 0 we might have lost changes due to recovery if client node isn't alive, e.g. powered off right after command was executed and return success but changes weren't committed yet on server.</p> <p>Mikhail, thanks, but in that case I don't understand the optimizations discussed by Robert and Andreas in bz15390. They claim the sync is only needed on directories, and only when the permissions are being reduced.</p> <p><a href="https://bugzilla.lustre.org/show_bug.cgi?id=15390" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=15390</a></p> <p>I made a patch according to the optimization proposal mentioned above: <a href="http://review.whamcloud.com/7257" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7257</a></p> <p>Did the patch landed fix this issue?</p> <p>The patch addresses the performance issue, but not the lack of documentation. I would like to see a step-by-step example scenario in which an asynchronous permission update leads to an inconsistent or insecure state. The example should explain why the problem is only relevant for directories. In other words, explain in precise terms the risks of disabling <tt>sync_permission</tt>.</p> <p>Ned,<br/> the specific problem that is being avoided here relates to Version Based Recovery (VBR), is a mechanism to allow some clients to recover from an MDS failure even in the (not so uncommon) case that one or more other active clients do not reconnect MDS and do their own recovery. Basic version-based recovery in the following situation:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># umask is 022, so dir1 is created with rwxr-xr-x permission client1$ mkdir /lustre/dir1 client2$ mkdir /lustre/dir2 client1$ chmod 700 /lustre/dir1 client3$ touch /lustre/dir1/file3 client4$ touch /lustre/dir1/file4 : : </pre> </div></div> <p>If client2 fails at the same time as the MDS (right after dir1 is created) and does not participate during MDS recovery, old Lustre recovery would prevent client<span class="error">&#91;34&#93;</span> from creating file<span class="error">&#91;34&#93;</span> because there would be a gap in the MDS transaction sequence, even though there is no dependency between these files and dir2. Similarly, if client1 failed, then file2 and file3 would not be able to recover, even if the dir1 creation was committed on the MDT before if crashed.</p> <p>With VBR, the replay for file3 and file4 would be dependent on the version of dir1 (transaction number in which dir1 was created/last modified), and not on each other. That would allow the files to be recreated from any running client, and only files created by the failing node would be lost.</p> <p>The <tt>sync_permission</tt> flag is concerned with avoiding the case where client1 fails after creating dir1 and running chmod, but the MDS only committed the mkdir and not the chmod before it fails. That would potentially allow the file creations to be replayed in a directory that does not have the correct permissions.</p> <p>Mike, thinking about this further, is the version of dir1 changed by the chmod so that the later file creates are dependent upon the new version of dir1 and not the old one? That would also prevent the later files to be created without any sync at all, though in most cases where permission changes are not being done this would increase the number of unreplayable RPCs in case of MDS failure. Could you please further clarify what specific problem the <tt>sync_permission</tt> behaviour is avoiding?</p> <p>Suggestion: That example (once the details have been confirmed) would make an excellent addition to the manual.</p> <p>Andreas, thanks for the explanation, but the pieces still aren't quite fitting together for me. In your example, dir1 perms change from 755 to 700, so non-owners lack write permission before and after the chmod. I'm not sure if that's what you intended.</p> <p>Given the optimization recently landed, we're only interested in the case where permissions are reduced. In that case, I imagine the spirit of the example is that <tt>touch</tt> would normally succeed <em>before</em> the <tt>chmod</tt>, but fail <em>after</em>. But if the <tt>chmod</tt> isn't replayed in recovery, client3 cannot distinguish that from the case where client1 never ran chmod (because it crashed), and therefore it is perfectly consistent to allow the touch to succeed. The directory would still have the old permissions which permitted file creation.</p> <p>I think I understand the point of the example now. The touch is predicated on the knowledge that chmod succeeded and the directory has secure permissions. If the chmod is lost, the directory contents become exposed to unauthorized users. I was assuming in my last comment that client3 couldn't see the results of the chmod until it was committed to disk, but now I suspect that it could. Do I have that right?</p> <p>Andreas, yes, chmod changed version of directory, so depended replays will be denied. But nothing prevents just regular creations in that directory after recovery, it will stay without permission for a while. So sync_permission flag is not just about recovery but more like policy when server guarantee all such changes to be done for sure, without sync_permission all works as before but permission might be lost, note this problem exists in all Lustre versions and is not about VBR. All further optimization are just attempts to find some balance between security and performance. </p> <p>Ned, your second assessment is correct, and I should have made that more clear in my comment. The danger is that the user thinks the chmod (or chown by root) succeeded, but it was lost during recovery, and this exposes files in the directory that shouldn't be visible to other users. Mike is correct that if file3/4 are created before MDS recovery (while the chmod/chown is still in the MDS cache) then they will be refused during replay because the dir1 permission change was lost when client1 failed and the creates will depend on the now-missing dir1 version. However, if file3/4 are created after MDS recovery (with chmod/chown lost) they will succeed.</p> <p>I guess <tt>sync_permissions</tt> is over &amp; above what POSIX requires even of a local filesystem, since a local filesystem would lose the chmod after a crash and the user might forget to re-do it, but at least there was a visible interruption to the user when the local node crashed. With a distributed filesystem, the failing node might be out of sight of the user.</p> <p>I filed <a href="https://jira.whamcloud.com/browse/LUDOC-180" title="documentation for the MDS sync_permission /proc tunable" class="issue-link" data-issue-key="LUDOC-180">LUDOC-180</a> to track the documentation for this /proc tunable, and this one can be closed since the patch to avoid sync operations for regular files and non-permission setattrs has landed for 2.5.0.</p> <p>It could potentially also be cherry-picked for 2.4.x and 2.1.x.</p> Duplicate LUDOC-180 Related LU-3660 LU-7239 Development Rank 1|hzvwpz: Rank (Obsolete) 9459 Severity