https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9929 Lustre <p>Hi ,</p> <p>When we setfacl default in lustre directory, I got unmapping group id(getfacl) after first setting(setfacl) . Then we executed setfacl command again and got fail ( Operation not permitted).</p> <p>Please help us to fix this problem. Thanks! </p> <p>The detail information is listed below.</p> <p>1.cat /etc/passwd<br/> user1:x:1001:1001::/home/user1:/bin/bash<br/> 2.nodemap setting<br/> nodemap.21b7e9f04fed448e.idmap=<br/> [<br/> ..... </p> { idtype: gid, client_id: 1001, fs_id: 23501 } <p>,<br/> .....<br/> ]<br/> 3.setfacl steps<br/> <span class="error">&#91;root@hsm client&#93;</span># mkdir hadoop3<br/> <span class="error">&#91;root@hsm client&#93;</span># getfacl /mnt/client/hadoop3<br/> getfacl: Removing leading '/' from absolute path names<br/> file: mnt/client/hadoop3<br/> owner: root<br/> group: root<br/> user::rwx<br/> group::r-x<br/> other::r-x<br/> <span class="error">&#91;root@hsm client&#93;</span># setfacl -R -d -m group:user1:rwx /mnt/client/hadoop3<br/> <span class="error">&#91;root@hsm client&#93;</span># getfacl /mnt/client/hadoop3<br/> getfacl: Removing leading '/' from absolute path names<br/> file: mnt/client/hadoop3<br/> owner: root<br/> group: root<br/> user::rwx<br/> group::r-x<br/> other::r-x<br/> default:user::rwx<br/> default:group::r-x<br/> default:group:23501:rwx<br/> default:mask::rwx<br/> default:other::r-x<br/> <span class="error">&#91;root@hsm client&#93;</span># setfacl -R -d -m group:user1:rwx /mnt/client/hadoop3<br/> setfacl: /mnt/client/hadoop3: Operation not permitted</p> Lustre 2.9 LU-9929

Use "setfacl" to set "default" setting fail when nodemap enabled

Bug Critical Resolved Fixed Emoly Liu sebg-crd-pm Wed, 30 Aug 2017 01:53:32 +0000 Thu, 12 Oct 2017 14:32:34 +0000 Sat, 30 Sep 2017 04:45:26 +0000 Lustre 2.11.0 Lustre 2.10.2 0 5 <p>Emoly</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Emoly,</p> <p>Do you have any comment about this issue? <br/> We really need your help. </p> <p>Thanks !</p> <p>I will look into this issue.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sebg-crd-pm" class="user-hover" rel="sebg-crd-pm">sebg-crd-pm</a>,<br/> I can't reproduce this issue in single node or multiple nodes tests. Here are my steps and output of my test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#On client node + groupadd -g 1001 user1 + useradd -g user1 -u 1001 user1 + cat /etc/passwd + grep user1 user1:x:1001:1001::/home/user1:/bin/bash #On MGS node + lctl nodemap_add nodemap_test + lctl nodemap_add_idmap --name nodemap_test --idtype gid --idmap 1001:23501 + lctl get_param 'nodemap.nodemap_test.*' nodemap.nodemap_test.admin_nodemap=0 nodemap.nodemap_test.deny_unknown=0 nodemap.nodemap_test.exports=[ ] nodemap.nodemap_test.fileset= nodemap.nodemap_test.id=5 nodemap.nodemap_test.idmap=[ { idtype: gid, client_id: 1001, fs_id: 23501 } ] nodemap.nodemap_test.map_mode=both nodemap.nodemap_test.ranges=[ ] nodemap.nodemap_test.squash_gid=99 nodemap.nodemap_test.squash_uid=99 nodemap.nodemap_test.trusted_nodemap=0 #On client node + mkdir -p /mnt/lustre/hadoop3 + getfacl /mnt/lustre/hadoop3 getfacl: Removing leading '/' from absolute path names # file: mnt/lustre/hadoop3 # owner: root # group: root user::rwx group::r-x other::r-x + echo '1st: setfacl' 1st: setfacl + setfacl -R -d -m group:user1:rwx /mnt/lustre/hadoop3 + getfacl /mnt/lustre/hadoop3 getfacl: Removing leading '/' from absolute path names # file: mnt/lustre/hadoop3 # owner: root # group: root user::rwx group::r-x other::r-x default:user::rwx default:group::r-x default:group:user1:rwx default:mask::rwx default:other::r-x + echo '2nd: setfacl' 2nd: setfacl + setfacl -R -d -m group:user1:rwx /mnt/lustre/hadoop3 + getfacl /mnt/lustre/hadoop3 getfacl: Removing leading '/' from absolute path names # file: mnt/lustre/hadoop3 # owner: root # group: root user::rwx group::r-x other::r-x default:user::rwx default:group::r-x default:group:user1:rwx default:mask::rwx default:other::r-x </pre> </div></div> <p>Could you provide the following information?</p> <ul class="alternate" type="square"> <li>Which nodes are there in your system and which nodes have user "user1"?</li> <li>Which steps did you run on which node?</li> <li>Can you provide all your nodemap information by the command "lctl get_param nodemap.$your_nodemap.*"?</li> <li>Can you collect some lustre logs on mgs node and client node during you test by the following commands: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#before test lctl set_param debug=-1 debug_mb=1000 lctl dk &gt; /dev/null #testing ... #after test lctl dk &gt; $logfile Then please upload the logfile here. </pre> </div></div></li> </ul> <ul class="alternate" type="square"> <li>What is your detailed lustre version?</li> </ul> <p>Thanks!</p> <p>Hi Emoly,</p> <p> Did you set your client ip to nodemap_test ? (lctl nodemap_add_range)?<br/> The client will in nodemap.default if you did not set it to nodemap_test.<br/> Please also check nodemap.active=1(lctl get_param nodemap.*) and "lctl set_param nodemap.nodemap_test.admin_nodemap=1" for root access permission. Thanks</p> <p>Which nodes are there in your system and which nodes have user "user1"?<br/> &gt;&gt; One server node 172.20.110.212(mgt/mdt/ost), One client node 172.20.110.211<br/> Which steps did you run on which node?<br/> &gt;&gt; Please see below<br/> Can you provide all your nodemap information by the command "lctl get_param nodemap.$your_nodemap.*"?<br/> &gt;&gt; Please see below <br/> Can you collect some lustre logs on mgs node and client node during you test by the following commands:<br/> &gt;&gt; see attached files.<br/> What is your detailed lustre version?<br/> &gt;&gt; Test it in Lustre 2.10 (lustre-release-58fd06e.tar.gz)/ one node(mgs/mds/oss ) follow these steps.</p> <p><span class="error">&#91;setup lustre&#93;</span><br/> lctl set_param nodemap.active=1<br/> lctl nodemap_add nodemap_test<br/> lctl set_param nodemap.nodemap_test.admin_nodemap=1<br/> lctl nodemap_add_idmap --name nodemap_test --idtype gid --idmap 1001:23501<br/> lctl nodemap_add_range --name nodemap_test --range 172.20.110.<span class="error">&#91;211-211&#93;</span>@o2ib<br/> lctl get_param nodemap.*<br/> lctl get_param nodemap.nodemap_test.*</p> <p><span class="error">&#91;output&#93;</span><br/> nodemap.active=1<br/> nodemap.nodemap_test.admin_nodemap=1<br/> nodemap.nodemap_test.deny_unknown=0<br/> nodemap.nodemap_test.exports=<br/> [<br/> </p> { nid: 172.20.110.211@o2ib, uuid: 2a980ffd-962a-eef8-37aa-cedf34253b31 } <p>, </p> { nid: 172.20.110.211@o2ib, uuid: 2a980ffd-962a-eef8-37aa-cedf34253b31 } <p>,<br/> ]<br/> nodemap.nodemap_test.fileset=</p> <p>nodemap.nodemap_test.id=1<br/> nodemap.nodemap_test.idmap=[</p> { idtype: gid, client_id: 1001, fs_id: 23501 } <p>]<br/> nodemap.nodemap_test.map_mode=both<br/> nodemap.nodemap_test.ranges=<br/> [</p> { id: 1, start_nid: 172.20.110.211@o2ib, end_nid: 172.20.110.211@o2ib } <p>]</p> <p><span class="error">&#91;client&#93;</span><br/> //client node user ppp: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28203/28203_aclserver.log" title="aclserver.log attached to LU-9929">aclserver.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28204/28204_aclclient.log" title="aclclient.log attached to LU-9929">aclclient.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> ppp:x:1001:1001::/home/ppp:/bin/bash</p> <p><span class="error">&#91;root@hsm mnt&#93;</span>mount.lustre 172.20.110.212@o2ib:/jlustre /mnt/lustre<br/> <span class="error">&#91;root@hsm mnt&#93;</span>mkdir -p /mnt/lustre/hadoop3<br/> <span class="error">&#91;root@hsm mnt&#93;</span># setfacl -R -d -m group:ppp:rwx /mnt/lustre/hadoop3<br/> <span class="error">&#91;root@hsm mnt&#93;</span># getfacl /mnt/lustre/hadoop3<br/> getfacl: Removing leading '/' from absolute path names</p> <ol> <li>file: mnt/lustre/hadoop3</li> <li>owner: root</li> <li>group: root<br/> user::rwx<br/> group::r-x<br/> other::r-x<br/> default:user::rwx<br/> default:group::r-x<br/> default:group:23501:rwx<br/> default:mask::rwx<br/> default:other::r-x</li> </ol> <p><span class="error">&#91;root@hsm mnt&#93;</span># setfacl -R -d -m group:ppp:rwx /mnt/lustre/hadoop3<br/> setfacl: /mnt/lustre/hadoop3: Operation not permitted</p> <p>Thanks, I can see this issue now. I will investigate it.</p> <p>The following log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:2.0:1504689607.481641:0:22011:0:(mdt_xattr.c:327:mdt_reint_setxattr()) Process leaving via out (rc=18446744073709551615 : -1 : 0xffffffffffffffff) </pre> </div></div> <p>shows the error comes from the following code</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> mdt_reint_setxattr() { ... rc = nodemap_map_acl(nodemap, rr-&gt;rr_eadata, xattr_len, NODEMAP_CLIENT_TO_FS); nodemap_putref(nodemap); <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out, rc); <span class="code-comment">/* ACLs were mapped out, <span class="code-keyword">return</span> an error so the user knows */</span> <span class="code-keyword">if</span> (rc != xattr_len) GOTO(out, rc = -EPERM); ... } </pre> </div></div> <p>The debugging information shows rc(44) != xattr_len(52). I will see what's wrong here.</p> <p>Hi Emoly, </p> <p>Do you have any update ?</p> <p>Thanks!</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sebg-crd-pm" class="user-hover" rel="sebg-crd-pm">sebg-crd-pm</a>,<br/> As you said, there are two issues:</p> <ul class="alternate" type="square"> <li>wrong mapping gid(getfacl): according to my following debugging information, the second tree_type should be 0 (NODEMAP_FS_TO_CLIENT) instead. This should be fixed soon. <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 13 15:03:02 centos7-2 kernel: id=1001, id_type=1, tree_type=1 Sep 13 15:03:02 centos7-2 kernel: id=23501, id_type=1, tree_type=1 </pre> </div></div></li> </ul> <ul class="alternate" type="square"> <li>EPERM(setfacl): the diff 52-44=8 bytes is caused by the acl entry whose nm_squash_id is 99. I still need some time to investigate where this squash id(99) comes from and why it needs to be skipped, and if we skip this check, why it returns EINVAL instead from ldiskfs.</li> </ul> <p>I will give a update later.</p> <p>When acl default unmapping code is added to mdt_getxattr(), EPERM issue disappears too.</p> <p>The issue happened because after the first setfacl, a wrong default acl was cached in the client side, when running setfacl again, FS didn't know this wrong unmapped gid(23501), so treated it as a squash id, then this squash id entry(8 bytes) was skipped in nodemap_map_acl(). That's why we saw EPERM error.</p> <p>I will submit a patch later.</p> <p>Emoly Liu (emoly.liu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29010" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29010</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9929" title="Use &quot;setfacl&quot; to set &quot;default&quot; setting fail when nodemap enabled" class="issue-link" data-issue-key="LU-9929"><del>LU-9929</del></a> nodemap: add unmapping process for default ACLs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0c7cc39d930d7ac7b2b0614d623103c5caddb2e6</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29010/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29010/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9929" title="Use &quot;setfacl&quot; to set &quot;default&quot; setting fail when nodemap enabled" class="issue-link" data-issue-key="LU-9929"><del>LU-9929</del></a> nodemap: add default ACL unmapping handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 62fee20556a4c90361bd28edb903dc77c9540133</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29369" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29369</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9929" title="Use &quot;setfacl&quot; to set &quot;default&quot; setting fail when nodemap enabled" class="issue-link" data-issue-key="LU-9929"><del>LU-9929</del></a> nodemap: add default ACL unmapping handling<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 6b798de0455d056d77ac5570187d3f550be52210</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29369/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29369/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9929" title="Use &quot;setfacl&quot; to set &quot;default&quot; setting fail when nodemap enabled" class="issue-link" data-issue-key="LU-9929"><del>LU-9929</del></a> nodemap: add default ACL unmapping handling<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 2ee62fbbf14e055d0134eb0859999be394909f8f</p> Duplicate LU-9759 Related Development Rank 1|hzzj9z: Rank (Obsolete) 9223372036854775807 Severity