https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16524 Lustre <p>We might need to support the use case of a 'local' admin, that is root on the client, also root on Lustre to achieve some tasks such as changing files' owner or group (so root squash cannot be used) but still restricted in some privileged actions (e.g. lfs commands).</p> LU-16524

Limit capabilities of local admin

Improvement Minor Resolved Fixed Sebastien Buisson Sebastien Buisson patch sec Thu, 2 Feb 2023 08:33:51 +0000 Mon, 5 Jun 2023 18:56:42 +0000 Sat, 20 May 2023 12:14:21 +0000 Lustre 2.16.0 Lustre 2.16.0 0 6 <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49873" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49873</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> nodemap: add rbac property to nodemap<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2b87dae7d926b0c244c75cde5cb0a1fac7c55a84</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49907</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> sec: enforce rbac roles<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fdd15c6a7af7bc0569b1d342c78b0d4b622ed7a5</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50184" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50184</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> sec: add fscrypt_admin rbac role<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5958dcbea1ded00b645da117f6f68cf86ba6168f</p> <p>here is mount failure after patch <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/50184/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/50184/</a></p> <p>nodemap configuration</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl nodemap_activate 0 lctl nodemap_del trusted lctl nodemap_del tenant100 lctl nodemap_modify --name default --property trusted --value 0 lctl nodemap_modify --name default --property admin --value 0 lctl nodemap_modify --name default --property deny_unknown --value 1 lctl nodemap_add trusted lctl nodemap_add_range --name trusted --range 192.168.200.[1-254]@tcp lctl nodemap_modify --name trusted --property trusted --value 1 lctl nodemap_modify --name trusted --property admin --value 1 lctl nodemap_modify --name trusted --property deny_unknown --value 0 lctl nodemap_add tenant100 lctl nodemap_add_range --name tenant100 --range 192.168.100.[1-254]@tcp lctl nodemap_modify --name tenant100 --property admin --value 0 lctl nodemap_activate 1 </pre> </div></div> <p>make sure nid belongs to nodemap "tenant100"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@server ~]# lctl nodemap_test_nid 192.168.100.2@tcp tenant100 </pre> </div></div> <p>On client (192.168.100.2@tcp)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@client:~# lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.100.2@tcp status: up interfaces: 0: enp5s0 root@client:~# mount -t lustre 192.168.200.2@tcp:/lustre /lustre mount.lustre: mount 192.168.200.2@tcp:/lustre at /lustre failed: Permission denied </pre> </div></div> <p>syslog shows below</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 7 05:28:59 client-tenant100 kernel: [3934814.817595] LustreError: 226554:0:(llite_lib.c:711:client_common_fill_super()) lustre-clilmv-ffff9df1ccbbe000: md_getattr failed for root: rc = -13 Mar 7 05:28:59 client-tenant100 kernel: [3934814.851720] Lustre: Unmounted lustre-client Mar 7 05:28:59 client-tenant100 kernel: [3934814.852581] LustreError: 226554:0:(super25.c:187:lustre_fill_super()) llite: Unable to mount &lt;unknown&gt;: rc = -13 </pre> </div></div> <p>Without patches, mount worked as expected even same nodemap policy applied.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@client:~# mount -t lustre 192.168.200.2@tcp:/lustre /lustre </pre> </div></div> <p>Thanks Shuichi for the heads-up.</p> <p>Could you please dump the whole nodemap configuration with this command, as I really need to know how it is setup?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param -R 'nodemap.*' </pre> </div></div> <p>In particular, could you please check that the squashed UID and GID do exist on client and server sides, with this command, run on both client and server sides?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># id &lt;squashed uid and gid&gt; </pre> </div></div> <p>Moreover, how is configured identity upcall?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param mdt.*-MDT*.identity_upcall </pre> </div></div> <p>Patch <a href="https://review.whamcloud.com/50184" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50184</a> is the topmost one in a series of 3 patches. When you say "without patches", does it mean you are rebuilding without just #50184, or you are also removing #49907 and #49873? By the way, are you using the tip of master branch? Can you please provide the reference to your current HEAD?<br/> It is also important for me to understand how the problem you see happens. Are you reformatting after rebuilding, and so starting with a brand new file system, or was it formatted before, and then it is upgraded?</p> <p>Thanks,<br/> Sebastien.</p> <p>This different behavior (not able to mount if root is squashed) changes was after patch 50184 applied.</p> <p>with patch 49907  here is lctl get_param -R 'nodemap.*' <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/48360/48360_lctl-get_param-nodemap-49907.txt" title="lctl-get_param-nodemap-49907.txt attached to LU-16524">lctl-get_param-nodemap-49907.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@server ~]# lctl get_param mdt.*-MDT*.identity_upcall mdt.lustre-MDT0000.identity_upcall=NONE mdt.lustre-MDT0001.identity_upcall=NONE root@client:~# id 99 id: ‘99’: no such user root@client:~# mount -t lustre 192.168.200.2@tcp:/lustre /lustre root@client:~# ls /lustre ls: cannot access '/lustre': Permission denied </pre> </div></div> <p>client was able to mount, it can't access filesystem.</p> <p>with patch 50184 lctl get_param -R 'nodemap.*' is attached <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/48361/48361_lctl-get_param-nodemap-50184.txt" title="lctl-get_param-nodemap-50184.txt attached to LU-16524">lctl-get_param-nodemap-50184.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@server ~]# lctl get_param mdt.*-MDT*.identity_upcall mdt.lustre-MDT0000.identity_upcall=NONE mdt.lustre-MDT0001.identity_upcall=NONE [root@server ~]# id 99 id: ‘99’: no such user root@client:~# id 99 id: ‘99’: no such user root@client:~# mount -t lustre 192.168.200.2@tcp:/lustre /lustre mount.lustre: mount 192.168.200.2@tcp:/lustre at /lustre failed: Permission denied </pre> </div></div> <p>client was not able to mount even identity_upcall=NONE.</p> <blockquote><p>It is also important for me to understand how the problem you see happens. Are you reformatting after rebuilding, and so starting with a brand new file system, or was it formatted before, and then it is upgraded?</p></blockquote> <p>When new build installed, I reformmated all OST/MDTs and re-applied nodemap setting below all time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl nodemap_activate 0 lctl nodemap_del trusted lctl nodemap_del tenant100 lctl nodemap_modify --name default --property trusted --value 0 lctl nodemap_modify --name default --property admin --value 0 lctl nodemap_modify --name default --property deny_unknown --value 1 lctl nodemap_add trusted lctl nodemap_add_range --name trusted --range 192.168.200.[1-254]@tcp lctl nodemap_modify --name trusted --property trusted --value 1 lctl nodemap_modify --name trusted --property admin --value 1 lctl nodemap_modify --name trusted --property deny_unknown --value 0 lctl nodemap_add tenant100 lctl nodemap_add_range --name tenant100 --range 192.168.100.[1-254]@tcp lctl nodemap_modify --name tenant100 --property admin --value 0 lctl nodemap_activate 1 </pre> </div></div> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50230" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50230</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> nodemap: filter out unknown records<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: cc28404fc05d36d7bdb55c032fd84adb84799511</p> <p>I have been testing that further, and found an interesting behavior. Without any <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> patches applied, it is not possible for a squashed root to mount the client if the squash uid or gid does not exist on server side, when SELinux is enabled on the client (either Permissive or Enforced). This is because the client will issue a <tt>getxattr</tt> request for <tt>security.selinux</tt>, and on server side user credentials are checked for a getxattr.</p> <p>With <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> patches applied (especially #50184), even when SELinux is not enabled on the client, it is not possible for a squashed root to mount the client if the squash uid or gid does not exist on server side. This is because patch #50184 adds a user credentials check on <tt>getattr</tt>.</p> <p>So I agree patch #50184 introduces a behavior change (only when SELinux is not enabled on the client), but I am not shocked to proceed to a user credentials check on <tt>getattr</tt>. Moreover, it is questionnable to mount a client as a mis-configured root (a squashed root with its squashed uid or gid that does not exist on server side), given that no further operation is possible on the file system. I would tend to think this is a nodemap configuration error and it should not be supported.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49873/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49873/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> nodemap: add rbac property to nodemap<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5e48ffca322c3c72d3b83b0719f245fc6f13c8e4</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49907/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49907/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> sec: enforce rbac roles<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 971e025f5fb77f4eaaa1e9070598dfa6292a9678</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50184/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50184/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> sec: add fscrypt_admin rbac role<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 22bef9b6c64ef394a2efb41ce1388be71300af0d</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50230/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50230/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16524" title="Limit capabilities of local admin" class="issue-link" data-issue-key="LU-16524"><del>LU-16524</del></a> nodemap: filter out unknown records<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 170ddaa96bdddea32a7c48ddacefc53d961cc783</p> <p>Sebastien, does it make sense for subdirectory mounts that the squashed root on the client is still able to chown/chmod/chgrp for files in the subdirectory tree <b>IF</b> they are for UID/GID/PROJID mapped in the nodemap (excluding root itself)? Similarly, it should only be possible for the squashed root to adjust quotas for UID/GID/PROJIDs that are mapped by the nodemap.</p> <p>That would allow the root user on the client to do normal admin tasks for files in that project/container, without needing to grant them "real" root access to the filesystem itself (i.e. <tt>admin=1</tt>).</p> <p>I think doing the mapped UID/GID/PROJID lookup is fast (this is already done for every RPC) so the only extra check would be whether it was the squashed root user on the client. Maybe an extra "<tt>squashed_admin=1</tt>" setting or similar?</p> <p>Is there anything left to do on this ticket, or should it be marked Resolved/Fixed? My last comment/question about restricting RBAC admin operations to IDs within the nodemap could be addressed in a separate ticket, though it would be nice to also get this into the 2.16 release so that the behavior is consistent.</p> <p>Hi Andreas,</p> <p>If I understand correctly your comment in <a href="https://jira.whamcloud.com/browse/LU-16524?focusedCommentId=370402&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-370402" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-16524?focusedCommentId=370402&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-370402</a>, that would consist in extending the capabilities of a squashed (root) user so that it can still modify file permissions and owners and quotas, <b>if</b> they correspond to a mapped uid/gid/projid.</p> <p>It seems to be quite the opposite idea of what we implemented with this ticket. The rbac roles are designed to limit the powers of not-squashed root, by preventing modifications of file permissions and owners (<tt>file_perms</tt> role), or quota modifications (<tt>quota_ops</tt> role) for instance.</p> <p>I am not saying that extending the capabilities of a squashed (root) user would not be an interesting feature to have. But I think it is a different approach that should be tackled under a different ticket.</p> <p>You are right, this would be somewhat the opposite approach, with the benefit that it would "grant" select privileges to the tenant admin, starting from "nothing" that the regular user has, so would be more "fail safe". The current approach will take away privileges from a root user, but risks that something was missed, or is added in the future that does not add RBAC roles/checks and cannot be squashed/removed.</p> <p>That said, it definitely belongs in a different ticket so that this one can be marked resolved..</p> <p>Seems like this body of work has merged for 2.16</p> Related Development Rank 1|i03brz: Rank (Obsolete) 9223372036854775807 Severity