https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1778 Lustre <p>On a node with root_squash activated, if root try to access to attributes of file (fstat) which has not been previously accessed, the operation return ENOPERM.<br/> If the attributes file were accessed by an authorized user, then root can access attributes without troubles.</p> <p>as root :<br/> <span class="error">&#91;root@clientae ~&#93;</span># mount -t lustre 192.168.1.100:/scratch /scratch<br/> <span class="error">&#91;root@clientae ~&#93;</span># cd /scratch/<br/> <span class="error">&#91;root@clientae scratch&#93;</span># ls -la<br/> total 16<br/> drwxrwxrwx 4 root root 4096 Aug 21 18:03 .<br/> dr-xr-xr-x. 28 root root 4096 Aug 22 15:53 ..<br/> drwxr-xr-x 2 root root 4096 Jun 21 18:42 .lustre<br/> drwx------ 2 slurm users 4096 Aug 21 18:03 test_dir<br/> <span class="error">&#91;root@clientae scratch&#93;</span># cd test_dir/<br/> <span class="error">&#91;root@clientae test_dir&#93;</span># ls -la<br/> ls: cannot open directory .: Permission denied</p> <p>then, as user 'slurm' :<br/> <span class="error">&#91;slurm@clientae ~&#93;</span>$ cd /scratch/test_dir<br/> <span class="error">&#91;slurm@clientae test_dir&#93;</span># ls -la<br/> total 16<br/> drwx------ 2 slurm users 4096 Aug 21 18:03 .<br/> drwxrwxrwx 4 root root 4096 Aug 22 16:47 ..<br/> <del>rw-r</del><del>r</del>- 1 slurm users 7007 Aug 22 15:58 afile</p> <p>now, come back as user root an replay the 'ls' command :<br/> <span class="error">&#91;root@clientae test_dir&#93;</span># ls -la<br/> total 16<br/> drwx------ 2 slurm users 4096 Aug 21 18:03 .<br/> drwxrwxrwx 4 root root 4096 Aug 22 16:47 ..<br/> <del>rw-r</del><del>r</del>- 1 slurm users 7007 Aug 22 15:58 afile<br/> <span class="error">&#91;root@clientae test_dir&#93;</span># stat afile<br/> File: `afile'<br/> Size: 7007 Blocks: 16 IO Block: 2097152 regular file<br/> Device: d61f715ah/3592384858d Inode: 144115238826934275 Links: 1<br/> Access: (0644/<del>rw-r</del><del>r</del>-) Uid: ( 500/ slurm) Gid: ( 100/ users)<br/> Access: 2012-08-22 15:59:26.000000000 +0200<br/> Modify: 2012-08-22 15:58:55.000000000 +0200<br/> Change: 2012-08-22 15:58:55.000000000 +0200</p> <p>At this point if you try to have a look into the file as root, you get ENOPERM<br/> <span class="error">&#91;root@clientae test_dir&#93;</span># cat afile<br/> cat: afile: Permission denied<br/> even if you already got access to the content with the authorized user.</p> <p>But, if the file is opened by the user ('tail -f afile' for exemple), root get access to the content of the file as well<br/> <span class="error">&#91;root@clientae test_dir&#93;</span># tail afile<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou<br/> coucou</p> <p>As soon as the file is closed by the user, root left access to the content(at least can't open the file any more)</p> <p>Alex.</p> LU-1778

Root Squash is not always properly enforced

Bug Minor Closed Fixed Niu Yawei Alexandre Louvet Wed, 22 Aug 2012 11:11:22 +0000 Tue, 28 Feb 2023 11:53:27 +0000 Fri, 9 May 2014 15:05:47 +0000 Lustre 2.1.1 Lustre 2.1.2 Lustre 2.6.0 Lustre 2.5.4 0 15 <p>Bob</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, any news on this ticket? Do you need some more information?</p> <p>I haven't been able to reproduce this failure in the current b2_1:</p> <p><span class="error">&#91;root@centos53 ~&#93;</span># mount -t lustre centos53:/lustre /mnt/lustre</p> <p><span class="error">&#91;root@centos53 ~&#93;</span># lctl get_param mdt/*/root_squash<br/> mdt.lustre-MDT0000.root_squash=500:500</p> <p><span class="error">&#91;root@centos53 ~&#93;</span># cd /mnt/lustre<br/> <span class="error">&#91;root@centos53 lustre&#93;</span># ls -la<br/> total 16<br/> drwxrwxrwx 4 root root 4096 Oct 1 09:06 .<br/> drwxr-xr-x. 6 root root 4096 Oct 1 09:05 ..<br/> drwx------ 2 bogl bogl 4096 Oct 1 09:07 bogl<br/> drwxr-xr-x 2 root root 4096 Oct 1 09:05 .lustre<br/> <span class="error">&#91;root@centos53 lustre&#93;</span># cd bogl<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># ls -la<br/> total 12<br/> drwx------ 2 bogl bogl 4096 Oct 1 09:07 .<br/> drwxrwxrwx 4 root root 4096 Oct 1 09:06 ..<br/> <del>rw</del>------ 1 bogl bogl 4 Oct 1 09:07 f1<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># cat f1<br/> foo</p> <p>Am I doing something incorrect in my reproduction attempt? Is there some other precondition to making this happen?</p> <p>This is worst than in my case ...</p> <p>1/ as root has been remapped to something different than 0:0, I would expect that you wasn't able to enter in bogl directory<br/> 2/ for the same reason, root shouldn't be able to watch the content of f1</p> <p>That said, I did a new test on a vanilla 2.1.3 (ie the rpm downloaded from whamcloud, without recompilation) on top of centos 6.x up to date to confirm that it fail at the latest available version.</p> <p><span class="error">&#91;root@server ~&#93;</span># lctl get_param mdt/*/root_squash<br/> mdt.scratch1-MDT0000.root_squash=0:0<br/> mdt.scratch2-MDT0000.root_squash=0:0<br/> mdt.scratch3-MDT0000.root_squash=0:0</p> <p>=&gt; set root_squash to an id which doesn't match my user id</p> <p><span class="error">&#91;root@server ~&#93;</span># lctl conf_param scratch1.mdt.root_squash="65535:65535"<br/> <span class="error">&#91;root@server ~&#93;</span># lctl get_param mdt/*/root_squash<br/> mdt.scratch1-MDT0000.root_squash=0:0<br/> mdt.scratch2-MDT0000.root_squash=0:0<br/> mdt.scratch3-MDT0000.root_squash=0:0</p> <p>On the client, running as a simple user<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ id<br/> uid=500(test) gid=100(users) groups=100(users)<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ pwd<br/> /scratch1<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ mkdir test<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ chmod 700 test<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ ls -la<br/> total 16<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 .<br/> dr-xr-xr-x. 29 root root 4096 Oct 2 09:22 ..<br/> drwxr-xr-x 2 root root 4096 Sep 11 22:15 .lustre<br/> drwx------ 2 test users 4096 Oct 2 10:37 test<br/> <span class="error">&#91;test@client scratch1&#93;</span>$ cd test/<br/> <span class="error">&#91;test@client test&#93;</span>$ echo coucou &gt; afile<br/> <span class="error">&#91;test@client test&#93;</span>$ ls -la<br/> total 9<br/> drwx------ 2 test users 4096 Oct 2 10:37 .<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 ..<br/> <del>rw-r</del><del>r</del>- 1 test users 7 Oct 2 10:37 afile<br/> <span class="error">&#91;test@client test&#93;</span>$ cat afile<br/> coucou</p> <p>now log as root on the client</p> <p><span class="error">&#91;root@client scratch1&#93;</span># pwd<br/> /scratch1<br/> <span class="error">&#91;root@client scratch1&#93;</span># ls -la<br/> total 16<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 .<br/> dr-xr-xr-x. 29 root root 4096 Oct 2 09:22 ..<br/> drwxr-xr-x 2 root root 4096 Sep 11 22:15 .lustre<br/> drwx------ 2 test users 4096 Oct 2 10:37 test<br/> <span class="error">&#91;root@client scratch1&#93;</span># cd test/<br/> <span class="error">&#91;root@client test&#93;</span># ls -la<br/> total 12<br/> drwx------ 2 test users 4096 Oct 2 10:37 .<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 ..<br/> <del>rw-r</del><del>r</del>- 1 test users 7 Oct 2 10:37 afile</p> <p>=&gt; There is already something funny at this point. As root was mapped to 65535:65535, I expect to not be able to enter in this directory (700) <span class="error">&#91;it was also shown in your test&#93;</span>. Flushing the cache on the client (ie echo 3 &gt; /proc/sys/vm/drop_caches make a different situation. Root can enter the 'test' directory, but can't stat files :</p> <p><span class="error">&#91;root@client test&#93;</span># ls -la<br/> ls: cannot access afile: Permission denied<br/> total 8<br/> drwx------ 2 test users 4096 Oct 2 10:37 .<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 ..<br/> -????????? ? ? ? ? ? afile</p> <p>I imagine this is due to the fact that the uid:gid translation in 'only' made at the mdt side and not at client side, letting root to access to attribute in client side cache without problem. Am I right ?</p> <p>Whatever, return as test user and stat the 'afile' again<br/> <span class="error">&#91;test@client test&#93;</span>$ ls -la<br/> total 12<br/> drwx------ 2 test users 4096 Oct 2 10:37 .<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 ..<br/> <del>rw-r</del><del>r</del>- 1 test users 7 Oct 2 10:37 afile</p> <p>switch back as root and run 'ls' once again let root again access to attributes :</p> <p><span class="error">&#91;root@client test&#93;</span># ls -la<br/> total 12<br/> drwx------ 2 test users 4096 Oct 2 10:37 .<br/> drwxrwxrwx 3 root root 4096 Sep 11 22:15 ..<br/> <del>rw-r</del><del>r</del>- 1 test users 7 Oct 2 10:37 afile</p> <p>at this point root can't access to 'afile' content<br/> <span class="error">&#91;root@client test&#93;</span># cat afile<br/> cat: afile: Permission denied</p> <p>unless an authorized user run tail -f afile, and keep it running<br/> <span class="error">&#91;test@client test&#93;</span>$ tail -f afile<br/> coucou</p> <p><span class="error">&#91;root@client test&#93;</span># cat afile <br/> coucou</p> <p>In my case id 500 == bogl. With root squash set to 500 (bogl) root should be able to see into bogl owned dir and file, and it does.</p> <p>I will retry with setting root squash to some other id.</p> <p>Have set root_squash to 65535:65535, shown by:</p> <p><span class="error">&#91;root@centos54 bogl&#93;</span># lctl set_param mdt/*/root_squash=65535:65535<br/> mdt.lustre-MDT0000.root_squash=65535:65535</p> <p>On client accessing as bogl, tree looks like:</p> <p><span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ ll -R /mnt/lustre<br/> /mnt/lustre:<br/> total 4<br/> drwx------ 2 bogl bogl 4096 Oct 1 15:18 bogl</p> <p>/mnt/lustre/bogl:<br/> total 4<br/> <del>rw</del>------ 1 bogl bogl 4 Oct 1 15:18 file<br/> <span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ <br/> <span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ cat /mnt/lustre/bogl/file<br/> foo</p> <p>Note permissions on dir and file only for bogl (id==500).</p> <p>Accessing as root, I consistently see no access for ls or file content to dir or file:</p> <p><span class="error">&#91;root@centos53 ~&#93;</span># ll -R /mnt/lustre<br/> /mnt/lustre:<br/> total 4<br/> drwx------ 2 bogl bogl 4096 Oct 1 15:18 bogl<br/> ls: cannot open directory /mnt/lustre/bogl: Permission denied<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># cat /mnt/lustre/bogl/file<br/> cat: /mnt/lustre/bogl/file: Permission denied</p> <p>I do see access being allowed for cd to the bogl owned dir. stat of the file is initially refused:</p> <p><span class="error">&#91;root@centos53 bogl&#93;</span># cd /mnt/lustre/bogl<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># stat file<br/> stat: cannot stat `file': Permission denied</p> <p>Then after doing a stat of the file as bogl:</p> <p><span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ stat /mnt/lustre/bogl/file<br/> File: `/mnt/lustre/bogl/file'<br/> Size: 4 Blocks: 8 IO Block: 2097152 regular file<br/> Device: 2c54f966h/743766374d Inode: 144115205255725058 Links: 1<br/> Access: (0600/<del>rw</del>------) Uid: ( 500/ bogl) Gid: ( 500/ bogl)<br/> Access: 2012-10-02 08:20:40.000000000 -0700<br/> Modify: 2012-10-01 15:18:07.000000000 -0700<br/> Change: 2012-10-01 15:18:46.000000000 -0700</p> <p>A later stat of the file as root is allowed:</p> <p><span class="error">&#91;root@centos53 bogl&#93;</span># stat file<br/> File: `file'<br/> Size: 4 Blocks: 8 IO Block: 2097152 regular file<br/> Device: 2c54f966h/743766374d Inode: 144115205255725058 Links: 1<br/> Access: (0600/<del>rw</del>------) Uid: ( 500/ bogl) Gid: ( 500/ bogl)<br/> Access: 2012-10-02 08:20:40.000000000 -0700<br/> Modify: 2012-10-01 15:18:07.000000000 -0700<br/> Change: 2012-10-01 15:18:46.000000000 -0700</p> <p>I see no case where access to the file content as root is allowed:</p> <p><span class="error">&#91;root@centos53 bogl&#93;</span># cat file<br/> cat: file: Permission denied<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># cat /mnt/lustre/bogl/file<br/> cat: /mnt/lustre/bogl/file: Permission denied</p> <p>This behavior looks consistent in all versions of 2.X right up to master.</p> <p>correction: on another retry I do see incorrect access to file content allowed. If I do a stat and then a persistent access as bogl:</p> <p><span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ stat /mnt/lustre/bogl/file<br/> File: `/mnt/lustre/bogl/file'<br/> Size: 4 Blocks: 8 IO Block: 2097152 regular file<br/> Device: 2c54f966h/743766374d Inode: 144115205255725058 Links: 1<br/> Access: (0600/<del>rw</del>------) Uid: ( 500/ bogl) Gid: ( 500/ bogl)<br/> Access: 2012-10-02 08:35:56.000000000 -0700<br/> Modify: 2012-10-01 15:18:07.000000000 -0700<br/> Change: 2012-10-01 15:18:46.000000000 -0700<br/> <span class="error">&#91;bogl@centos53 lustre-release&#93;</span>$ tail -f /mnt/lustre/bogl/file<br/> foo</p> <p>After that a stat and access as root is allowed, at least for a while:</p> <p><span class="error">&#91;root@centos53 bogl&#93;</span># stat file<br/> File: `file'<br/> Size: 4 Blocks: 8 IO Block: 2097152 regular file<br/> Device: 2c54f966h/743766374d Inode: 144115205255725058 Links: 1<br/> Access: (0600/<del>rw</del>------) Uid: ( 500/ bogl) Gid: ( 500/ bogl)<br/> Access: 2012-10-02 08:50:02.000000000 -0700<br/> Modify: 2012-10-01 15:18:07.000000000 -0700<br/> Change: 2012-10-01 15:18:46.000000000 -0700<br/> <span class="error">&#91;root@centos53 bogl&#93;</span># cat file<br/> foo</p> <p>It seems to require both a (permitted) stat and file access as bogl before the access that should be forbidden as root gets allowed.</p> <p>Niu is going to look into this one</p> <p>Hi, Alex</p> <p>As you mentioned, root_suqash is just a server side id remapping (like NFS root_squash), it doesn't affect client cache, so this looks an expected behaviour to me. You need to make sure the cache is cleared before you want the root_squash enforced. (I think it's same for NFS, isn't it?) Thanks.</p> <p>Hi Niu,</p> <p>I made some test with NFS and it works as expected : root (under root_squash) never get access to user data if rights for 'others' are not set. It doesn't depend of the activity of an authorized user on the same client.<br/> I should add that I can't make sure the cache is cleared before we want the root_squash enforced : root_squash is expectected to be enforced all the time and the cache content depend of the authorized user activity.</p> <blockquote> <p>I made some test with NFS and it works as expected : root (under root_squash) never get access to user data if rights for 'others' are not set. It doesn't depend of the activity of an authorized user on the same client.</p></blockquote> <p>I think NFS client should not know if server is squashing root as well, but there are several reasons that I can think of, which could make NFS root_squash doesn't affected by the client cache much:</p> <ul class="alternate" type="square"> <li>NFS is kind of WCC (Weak Cache Consistency) filesystem, it revalidates client cache every few seconds (default is 3?), and with some mount options, the cache could be revalidated before every operation.</li> </ul> <ul class="alternate" type="square"> <li>NFS client sends ACCESS RPC for root user before operations.</li> </ul> <p>Lustre is a strong cache consistency filesystem, we can't afford the extra ACCESS RPC or cache revalidation like NFS does, maybe we can make the client aware of the root_squash setting on server, and let the user to configure if they want always do access check on server side (with sacrificing the performance), but I'm not sure if we have enough resource to implement it for this moment. Anyway, I think we should state the root_squash on caching problem clearly in the manual.</p> <p>Alex, what do you think about? Is this feature (enforce root_squash no matter of caching) very important for you? Or just improving manual is ok for you? Thanks.</p> <p>Hi Niu,</p> <p>We do not ask that setting or unsetting<br/> the root_squash parameter is taken into account on the whole Lustre cluster in real time. We could really accommodate with unmounting then remounting the clients if the root_squash parameter has changed on the<br/> server.<br/> Our real issue is that the root user accessing a file from a client where the same file has already been accessed by a legitimate user will gain access to this file, whatever is the root_squash parameter, because data will be read from the client cache.<br/> I think it could be possible to store the root_squash information on the client at mount time. So there would be no need to verify this on the server for every request, and there would be no impact on performance.</p> <p>What do you think?</p> <p>Sebastien.</p> <p>Hi, Sebastien</p> <p>Yes, I agree with you on this. Adding a permission checking hook (and check the squash setting there) for llite and make the llite aware of root_squash setting could save the RPCs to server.</p> <p>In my opinion, this could be a feature enhancement rather than a bug, I'm glad to implement this when the time is available, and if you want propose a patch for this, I'm glad to help on review. Thank you.</p> <p>Hi,</p> <p>I would like to propose a patch to address this issue, so I carried out some tests to try to understand which functions are involved in getting file permissions and granting or not file access.<br/> Unfortunately, my tests left me a little bit confused...</p> <p>Here is what I did.<br/> File owner is user buisso1s. root_squash is enforced.</p> <ul class="alternate" type="square"> <li>accessing as user pichong:<br/> <del>EACCES in ll_file_open() (file</del>&gt;private_data is not NULL)</li> <li>accessing as root:<br/> <del>EACCES in ll_file_open() (file</del>&gt;private_data is not NULL)</li> <li>accessing as user pichong, while buisso1s runs 'tail -f file':<br/> -EACCES in ll_inode_permission()</li> <li>accessing as root, while buisso1s runs 'tail -f file':<br/> Access granted, both ll_file_open() and ll_inode_permission() return 0 (file-&gt;private_data is NULL)</li> </ul> <p>In the end, I could not figure out who is in charge of checking file permissions.<br/> Can you shed my light on this?</p> <p>TIA,<br/> Sebastien.</p> <p>Hi,</p> <p>When there isn't cache on client, permission checking is done on server side on the open RPC(the first two cases), when there is cache on client (no open RPC is needed), permission check is done on client by the permisson checking hook (ll_inode_permission, which is invoked by kernel, see may_open()).</p> <p>So, is it OK if I propose to modify ll_inode_permission() to add a check for some kind of root_squash parameter that would be fetched by the client at mount time and stored somewhere?</p> <p>Hi, Sebastien, I think it's doable. The current root_squash option is stored in mdt config log (because it's a mds only option), we could probably populate this option into the client config log as well, then the client can be notified whenever the option is changed. Thanks.</p> <p>Hi,<br/> I think I need some help regarding the way to store the root_squash option in the client config log. At the moment this is stored in the mdt config log, so how is it possible to pass it to the client config log? Via the mgs config log? What are the functions involved in that case?<br/> TIA,<br/> Sebastien.</p> <p>Hi, Sebastien</p> <p>Please look at the mgs_write_log_param(), root_squash is now a PARAM_MDT param, which is stored in the $FSNAME-mdt0001, you might want it be stored in the client log either ($FSNAME-client), I think a simple way is to have the amdinistrator runnig two configure commands:<br/> 1. lctl conf_param $FSNAME.mdt.rootsquash=$ID:$ID<br/> 2. lctl conf_param $FSNAME.llite.rootsquash=$ID:$ID</p> <p>And the other options related to root_squash should be treated carefully as well, such as nosquash_nids. Thanks.</p> <p>I have posted a patch for b2_1 on gerrit: <a href="http://review.whamcloud.com/5212" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5212</a></p> <p>For information, here is the note from Andreas Dilger in the gerrit.</p> <blockquote> <p>Patch Set 1: I would prefer that you didn't submit this</p> <p>I don't think this patch introduces any useful security to the system. If the user is root on the client, then it is trivial to "su" to another user and bypass the client-side root squash entirely.</p></blockquote> <p>This is also true for NFS, but this is not the problem. Lustre claim to support root_squash (at least there is a chapter in the documentation about this feature) and customer hope that this functionality avoid root to access files for which root user doesn't have access. I agree that root can modify it credential and access to the file, but this is another story.</p> <p>The only real interest of this feature is to avoid root to make stupid actions that will damage the content of the filesystem, but the comportment of the feature should be consistent over time and not change due to the client state. Currently root_squash comportment is confusing and the request is just to make it clean.</p> <p>Excerpt from Andreas comment in the gerrit</p> <blockquote> <p>...<br/> In summary, there is absolutely nothing to be gained except code complexity if the user already has root access on the client. This has to be enforced at the server, and at most root squash can only prevent the user from accessing files owned by root in the filesystem, or other root-only operations.</p> <p>Only with Kerberos and/or the upcoming UID/GID mapping could root be denied access to <em>new</em> files from that client, and I can't think of any way that root could be denied access to cached files on the client. Even if the user's keys were only in memory and the kernel itself blocked access from root locally (in an irrevocable manner), the root user could replace the lustre kernel modules with an insecure version and reboot, and then wait until the user accessed secure data again.</p> <p>The only way to avoid this is to never allow root access on the client in the first place.</p></blockquote> <p>Andreas,</p> <p>The current implementation of root squash feature by Lustre is not working as expected by the customer and as specified in "Using Root Squash" section of the Lustre Operations Manual.</p> <p>What do you propose to make progress on this issue ?</p> <p>If you think this feature is senseless, then why not reducing its scope to security configurations only (MDT sec-level), or even remove the feature completely ?</p> <p>My feeling is that we should be able to make it work properly. We could perform the root squashing on the client by overwritting the fsuid and gsuid of the task with the root_squash uid:guid specified on the MDS. These settings could be transmitted to the client either at mount time or each time file attributes are retrieved from the MDS (LDLM_INTENT_OPEN or LDLM_INTENT_GETATTR rpcs for instance). The patch I proposed last week is not suitable. Ok, let's find a better solution.</p> <p>Actually, the description in the use manual is correctly describing how the code functions:</p> <p><a href="http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438221_64726" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438221_64726</a></p> <blockquote> <p>Root squash is a security feature which restricts super-user access rights to a Lustre file system. Without the root squash feature enabled, Lustre users on untrusted clients could access or modify files owned by root on the filesystem, including deleting them. Using the root squash feature restricts file access/modifications as the root user to only the specified clients. Note, however, that <em>this does not prevent users on insecure clients from accessing files owned by other users</em>.</p></blockquote> <p>Like I wrote in the Gerrit comment, there is nothing that can be done by root squash to prevent access to files when someone has root access on the client. In that case, the root user could "<tt>su - other_user</tt>" and immediately circumvent all of the checking that was added to squash the root user access. The root squash feature is only to prevent "root" on clients to be able to access and/or modify files <em>owned by root</em> on the filesystem. The same "<tt>su - other_user</tt>" hole is present for NFS, and the fact that "root" is denied direct access on NFS is like a sheet of paper protecting a bank vault.</p> <p>The OpenSFS UID/GIF mapping and shared-key authentication features being developed by IU could allow for much more robust protection in the future. This would allow mapping users from specific nodes to one set of UIDs that don't overlap with UIDs from other nodes, and with the shared-key node authentication it would be impossible for even root to access files for UIDs that are not mapped to that cluster. If you are interested to follow this design and development, please email me and I will provide meeting and list details.</p> <p>Andreas, I think we are moving away from this ticket objective. I do agree with all points about security limitations of the root_squash feature, but this is not the problem there. The problem is that the manual says that access to root user is only granted to object for which it is allowed and this is not <b>always</b> true.</p> <p>Case for which root try to get read access to object for witch the inode is already in client cache, doesn't get root_squash applied. Client code doesn't have knowledge about root_squash and only apply traditional security checking. The result is that root get access granted or denied depending of the cache content, which is very confusing for users. This is the only reason on the jira ticket.</p> <p>I have posted a patch for master on gerrit: <a href="http://review.whamcloud.com/#change,5700" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5700</a></p> <p>Tests on the patchset 7 and 8 have made the client hang after conf-sanity test_43 (the one for root squash). I have been able to reproduce the hang (after 16 successful runs) and took a dump.</p> <p>It is available on ftp.whamcloud.com in /uploads/<a href="https://jira.whamcloud.com/browse/LU-1778" title="Root Squash is not always properly enforced" class="issue-link" data-issue-key="LU-1778"><del>LU-1778</del></a><br/> ftp&gt; dir<br/> 227 Entering Passive Mode (72,18,218,227,205,178).<br/> 150 Here comes the directory listing.<br/> <del>rw-r</del><del>r</del>- 1 123 114 3387608 Jul 25 07:22 lustre-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 45824206 Jul 25 07:23 lustre-debuginfo-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 181316 Jul 25 07:23 lustre-ldiskfs-4.1.0-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 1674715 Jul 25 07:23 lustre-ldiskfs-debuginfo-4.1.0-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 3312152 Jul 25 07:23 lustre-modules-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 165060 Jul 25 07:24 lustre-osd-ldiskfs-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 5067172 Jul 25 07:24 lustre-source-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 4757320 Jul 25 07:24 lustre-tests-2.4.51-2.6.32_358.el6.x86_64_g4c66dbd.x86_64.rpm<br/> <del>rw-r</del><del>r</del>- 1 123 114 100181834 Jul 25 07:27 vmcore</p> <p>Here are the information I have extracted from the dump.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> The unmount command seems hung. Higher part of the stack is due to the dump signal. crash&gt; bt 2723 PID: 2723 TASK: ffff88046ab98040 CPU: 4 COMMAND: "umount" #0 [ffff880028307e90] crash_nmi_callback at ffffffff8102d2c6 #1 [ffff880028307ea0] notifier_call_chain at ffffffff815131d5 #2 [ffff880028307ee0] atomic_notifier_call_chain at ffffffff8151323a #3 [ffff880028307ef0] notify_die at ffffffff8109cbfe #4 [ffff880028307f20] do_nmi at ffffffff81510e9b #5 [ffff880028307f50] nmi at ffffffff81510760 [exception RIP: page_fault] RIP: ffffffff815104b0 RSP: ffff880472c13bc0 RFLAGS: 00000082 RAX: ffffc9001dd57008 RBX: ffff880470b27e40 RCX: 000000000000000f RDX: ffffc9001dd1d000 RSI: ffff880472c13c08 RDI: ffff880470b27e40 RBP: ffff880472c13c48 R8: 0000000000000000 R9: 00000000fffffffe R10: 0000000000000001 R11: 5a5a5a5a5a5a5a5a R12: ffff880472c13c08 = struct cl_site * R13: 00000000000000c4 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #6 [ffff880472c13bc0] page_fault at ffffffff815104b0 #7 [ffff880472c13bc8] cfs_hash_putref at ffffffffa04305c1 [libcfs] #8 [ffff880472c13c50] lu_site_fini at ffffffffa0588841 [obdclass] #9 [ffff880472c13c70] cl_site_fini at ffffffffa0591d0e [obdclass] #10 [ffff880472c13c80] ccc_device_free at ffffffffa0e6c16a [lustre] #11 [ffff880472c13cb0] lu_stack_fini at ffffffffa058b22e [obdclass] #12 [ffff880472c13cf0] cl_stack_fini at ffffffffa059132e [obdclass] #13 [ffff880472c13d00] cl_sb_fini at ffffffffa0e703bd [lustre] #14 [ffff880472c13d40] client_common_put_super at ffffffffa0e353d4 [lustre] #15 [ffff880472c13d70] ll_put_super at ffffffffa0e35ef9 [lustre] #16 [ffff880472c13e30] generic_shutdown_super at ffffffff8118326b #17 [ffff880472c13e50] kill_anon_super at ffffffff81183356 #18 [ffff880472c13e70] lustre_kill_super at ffffffffa057d37a [obdclass] #19 [ffff880472c13e90] deactivate_super at ffffffff81183af7 #20 [ffff880472c13eb0] mntput_no_expire at ffffffff811a1b6f #21 [ffff880472c13ee0] sys_umount at ffffffff811a25db #22 [ffff880472c13f80] system_call_fastpath at ffffffff8100b072 RIP: 00007f0e6a971717 RSP: 00007fff17919878 RFLAGS: 00010206 RAX: 00000000000000a6 RBX: ffffffff8100b072 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f0e6c3cfb90 RBP: 00007f0e6c3cfb70 R8: 00007f0e6c3cfbb0 R9: 0000000000000000 R10: 00007fff179196a0 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 00007f0e6c3cfbf0 ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b ccc_device_free() is called on lu_device 0xffff880475ad06c0 crash&gt; struct lu_device ffff880475ad06c0 struct lu_device { ld_ref = { counter = 1 }, ld_type = 0xffffffffa0ea22e0, ld_ops = 0xffffffffa0e787a0, ld_site = 0xffff880472cf05c0, ld_proc_entry = 0x0, ld_obd = 0x0, ld_reference = {&lt;No data fields&gt;}, ld_linkage = { next = 0xffff880472cf05f0, prev = 0xffff880472cf05f0 } } ld_type-&gt;ldt_tags crash&gt; rd -8 ffffffffa0ea22e0 ffffffffa0ea22e0: 04 = LU_DEVICE_CL ld_type-&gt;ldt_name crash&gt; rd ffffffffa0ea22e8 ffffffffa0ea22e8: ffffffffa0e7d09f = "vvp" lu_site=ffff880472cf05c0 crash&gt; struct lu_site ffff880472cf05c0 struct lu_site { ls_obj_hash = 0xffff880470b27e40, ls_purge_start = 0, ls_top_dev = 0xffff880475ad06c0, ls_bottom_dev = 0x0, ls_linkage = { next = 0xffff880472cf05e0, prev = 0xffff880472cf05e0 }, ls_ld_linkage = { next = 0xffff880475ad06f0, prev = 0xffff880475ad06f0 }, ls_ld_lock = { raw_lock = { slock = 65537 } }, ls_stats = 0xffff880470b279c0, ld_seq_site = 0x0 } crash&gt; struct cfs_hash 0xffff880470b27e40 struct cfs_hash { hs_lock = { rw = { raw_lock = { lock = 0 } }, spin = { raw_lock = { slock = 0 } } }, hs_ops = 0xffffffffa05edee0, hs_lops = 0xffffffffa044e320, hs_hops = 0xffffffffa044e400, hs_buckets = 0xffff880471e4f000, hs_count = { counter = 0 }, hs_flags = 6184, = 0x1828 = CFS_HASH_SPIN_BKTLOCK | CFS_HASH_NO_ITEMREF | CFS_HASH_ASSERT_EMPTY | CFS_HASH_DEPTH hs_extra_bytes = 48, hs_iterating = 0 '\000', hs_exiting = 1 '\001', hs_cur_bits = 23 '\027', hs_min_bits = 23 '\027', hs_max_bits = 23 '\027', hs_rehash_bits = 0 '\000', hs_bkt_bits = 15 '\017', hs_min_theta = 0, hs_max_theta = 0, hs_rehash_count = 0, hs_iterators = 0, hs_rehash_wi = { wi_list = { next = 0xffff880470b27e88, prev = 0xffff880470b27e88 }, wi_action = 0xffffffffa04310f0 &lt;cfs_hash_rehash_worker&gt;, wi_data = 0xffff880470b27e40, wi_running = 0, wi_scheduled = 0 }, hs_refcount = { counter = 0 }, hs_rehash_buckets = 0x0, hs_name = 0xffff880470b27ec0 "lu_site_vvp" } </pre> </div></div> <p>I am going to attach the log of the Maloo test that hung (Jul 19 10:12 PM).</p> <p>client log from Maloo test on patchset 7 (Jul 19 10:12 PM)</p> <p>I have posted another patch that adds a service to print a nidlist: <a href="http://review.whamcloud.com/#/c/8479/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8479/</a> . After review of patchset 11 of #5700 patch, it seems to be a requirement.</p> <p>Thank you. Would it be possible for you to rebase this on current master? There are a few conflicts preventing merge. </p> <p>The patch #8479 has been landed and then reverted due to a conflit with GNIIPLND patch.</p> <p>I have posted a new version of the patch: <a href="http://review.whamcloud.com/9221" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9221</a></p> <p>Patch landed to Master</p> <p>This ticket has not been fixed yet.<br/> The main patch <a href="http://review.whamcloud.com/#change,5700" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5700</a> is still in progress.</p> <p>Now really landed for 2.6. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I have backported the two patches to be integrated in 2.5 maintenance release.<br/> <a href="http://review.whamcloud.com/10743" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10743</a><br/> <a href="http://review.whamcloud.com/10744" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10744</a></p> <p>The two above patches #10743 and #10744 have been posted and are ready for review since end of June.<br/> Would it be possible to have them included in the next 2.5 maintenance release: 2.5.3 ?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/10743/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10743/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1778" title="Root Squash is not always properly enforced" class="issue-link" data-issue-key="LU-1778"><del>LU-1778</del></a> libcfs: add a service that prints a nidlist<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 57a8a6bec4dc965388b5bba48e7501f79bdab44b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/10744/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10744/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1778" title="Root Squash is not always properly enforced" class="issue-link" data-issue-key="LU-1778"><del>LU-1778</del></a> llite: fix inconsistencies of root squash feature<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: d82b4f54cbbe269519330e88639dd8e197636496</p> <p>Closing as the issue has been fixed (several months ago) in master and 2.5 maintenance release.</p> Blocker Related LU-5142 LU-6990 Development Rank 1|hzvskf: Rank (Obsolete) 8532 Severity