https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16054 Lustre <p>We have a client accessing a file system via a nodemap.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ubuntu@lus2526-tcp1:~$ cat /sys/fs/lustre/version  2.14.0_2_gb280f22 ubuntu@lus2526-tcp1:~$ df /lustre/scratch12{5..6} Filesystem                                     1K-blocks  Used     Available Use% Mounted on 10.160.40.37@tcp1:10.160.40.36@tcp1:/lus25 5161226796192 21176 5109139362568   1% /lustre/scratch125 10.160.42.37@tcp1:10.160.42.36@tcp1:/lus26 4301022330160 17604 4257616135516   1% /lustre/scratch126 ubuntu@lus2526-tcp1:~$ id uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),117(netdev),118(lxd) ubuntu@lus2526-tcp1:~$ echo &gt; /lustre/scratch125/$(uname -n) ; echo &gt; /lustre/scratch126/$(uname -n) ubuntu@lus2526-tcp1:~$ ls -l /lustre/scratch12*/* -rw-rw-r-- 1 ubuntu ubuntu 1 Jul 28 07:45 /lustre/scratch125/lus2526-tcp1 -rw-rw-r-- 1 ubuntu ubuntu 1 Jul 28 07:45 /lustre/scratch126/lus2526-tcp1</pre> </div></div> <p>The external default client sees:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> jb23@gen3-os0000011:~$ id jb23 uid=12296(jb23) gid=1105(team94) groups=1105(team94),15283(sag-secure-fileshare),1415(ssg-confluence),1400(www-pagesmith),15016(isg-dcim),4999(jb23test),15141(sag-mfa),1533(ssg-isg),1490(docker),706(ssg),15404(dir-admins),15456(IDS),15264(sag-mso365-apps) jb23@gen3-os0000011:~$ find  /lustre/scratch12[56]/admin/team94/jb23/tcp* -type f -ls 144115339507007490      4 -rw-rw-r--   1 99       acedbdoc        1 Jul 28 08:45 /lustre/scratch125/admin/team94/jb23/tcp1/lus2526-tcp1 144115406599094274      4 -rw-rw-r--   1 99       acedbdoc        1 Jul 28 08:45 /lustre/scratch126/admin/team94/jb23/tcp1/lus2526-tcp1 jb23@gen3-os0000011:~$ ls -l /lustre/scratch12*/admin/team94/jb23/tcp1/lus2526-tcp1 -rw-rw-r-- 1 99 acedbdoc 1 Jul 28 08:45 /lustre/scratch125/admin/team94/jb23/tcp1/lus2526-tcp1 -rw-rw-r-- 1 99 acedbdoc 1 Jul 28 08:45 /lustre/scratch126/admin/team94/jb23/tcp1/lus2526-tcp1 jb23@gen3-os0000011:~$ getent group acedbdoc acedbdoc:*:99:image jb23@gen3-os0000011:~$ ls -l .bashrc -rw-r--r-- 1 jb23 team94 1130 May 26 12:47 .bashrc </pre> </div></div> <p> </p> <p>The nodemap is configured</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus25-mds1 ~]# cat /sys/fs/lustre/version  2.12.6_ddn66 [root@lus25-mds1 lustre_casm16]# <span class="code-keyword">for</span> i in * &gt; <span class="code-keyword">do</span> &gt; echo $i $(cat $i ) &gt; done admin_nodemap 0 audit_mode 1 deny_unknown 0 exports [ { nid: 10.177.127.35@tcp1, uuid: 720c5dc0-efbf-4d55-9340-8a2bde26d039 }, ] fileset /admin/team94/jb23/tcp1 id 1 idmap [ { idtype: uid, client_id: 1000, fs_id: 12296 }, { idtype: gid, client_id: 1000, fs_id: 1105 } ] map_mode all ranges [ { id: 1, start_nid: 10.177.126.0@tcp1, end_nid: 10.177.127.255@tcp1 } ] sepol squash_gid 1105 squash_projid   squash_uid 12296 trusted_nodemap 0</pre> </div></div> LU-16054

changes in lu-14797 create issues with older clients accessing fs though nodemaps

Bug Minor Open Unresolved Sebastien Buisson James Beal Thu, 28 Jul 2022 10:49:51 +0000 Fri, 29 Sep 2023 14:37:41 +0000 0 5 <p>root@gen3-os0000011:~# ls -l  /lustre/scratch125/admin/team94/<br/> total 4<br/> drwxr-xr-x 18 99 acedbdoc 4096 Jul 21 14:39 jb23<br/> root@gen3-os0000011:~# ls -l  /lustre/scratch125/admin/team94<br/> total 4<br/> drwxr-xr-x 18 99 acedbdoc 4096 Jul 21 14:39 jb23<br/> root@gen3-os0000011:~# ls -l  /lustre/scratch125/admin<br/> total 4<br/> drwxr-xr-x 3 99 acedbdoc 4096 Jul 19 15:07 team94<br/> root@gen3-os0000011:~# ls -l  /lustre/scratch125<br/> total 4<br/> drwxr-xr-x 3 99 acedbdoc 4096 Jul 19 15:07 admin<br/> It looked like all the files/directories have had their owner/group changed.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> root@gen3-os0000011:~# ls -ld /lustre/scratch125/admin/team94/jb23/lfs_test drwxrwxrwx 2 99 acedbdoc 4096 Jul 27 15:30 /lustre/scratch125/admin/team94/jb23/lfs_test root@gen3-os0000011:~# chown jb23 /lustre/scratch125/admin/team94/jb23/lfs_test chown: changing ownership of <span class="code-quote">'/lustre/scratch125/admin/team94/jb23/lfs_test'</span>: Operation not permitted root@gen3-os0000011:~# </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> root@gen3-os0000011:~# lctl list_nids 172.27.71.121@tcp</pre> </div></div> <p>the default nodemap appears to have been changed, this may be unrelated however its not something I have done delibrately. And are other system we are commissioning has the same issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus25-mds1 ost-survey]# lctl nodemap_test_nid 172.27.71.121@tcp <span class="code-keyword">default</span> [root@lus25-mds1 exports]# lctl get_param -R <span class="code-quote">'nodemap.<span class="code-keyword">default</span>'</span>  nodemap.<span class="code-keyword">default</span>.admin_nodemap=0 nodemap.<span class="code-keyword">default</span>.audit_mode=1 nodemap.<span class="code-keyword">default</span>.exports= [  { nid: 10.177.161.188@tcp5, uuid: 12ebc53b-9e45-427d-8e30-5a2439569728 }, { nid: 172.27.71.121@tcp, uuid: a64511ef-8ced-4702-892e-50a74c631d98 }, { nid: 10.160.40.12@tcp, uuid: lus25-MDT0000-lwp-OST0008_UUID }, { nid: 10.160.40.12@tcp, uuid: lus25-MDT0000-lwp-OST0009_UUID }, { nid: 10.160.40.13@tcp, uuid: lus25-MDT0000-lwp-OST000a_UUID }, { nid: 10.160.40.10@tcp, uuid: lus25-MDT0000-lwp-OST0005_UUID }, { nid: 10.160.40.11@tcp, uuid: lus25-MDT0000-lwp-OST0006_UUID }, { nid: 10.160.40.9@tcp, uuid: lus25-MDT0000-lwp-OST0002_UUID }, { nid: 10.160.40.8@tcp, uuid: lus25-MDT0000-lwp-OST0001_UUID }, { nid: 10.160.40.9@tcp, uuid: lus25-MDT0000-lwp-OST0003_UUID }, { nid: 10.160.40.8@tcp, uuid: lus25-MDT0000-lwp-OST0000_UUID }, { nid: 10.160.40.5@tcp, uuid: lus25-MDT0000-lwp-MDT0001_UUID }, { nid: 10.160.40.5@tcp, uuid: lus25-MDT0001-mdtlov_UUID }, { nid: 0@lo, uuid: lus25-MDT0000-lwp-MDT0000_UUID }, { nid: 10.160.40.6@tcp, uuid: lus25-MDT0000-lwp-MDT0002_UUID }, { nid: 10.160.40.6@tcp, uuid: lus25-MDT0002-mdtlov_UUID }, { nid: 10.160.40.7@tcp, uuid: lus25-MDT0000-lwp-MDT0003_UUID }, { nid: 10.160.40.7@tcp, uuid: lus25-MDT0003-mdtlov_UUID }, { nid: 10.160.40.13@tcp, uuid: lus25-MDT0000-lwp-OST000b_UUID }, { nid: 10.160.40.10@tcp, uuid: lus25-MDT0000-lwp-OST0004_UUID }, { nid: 10.160.40.11@tcp, uuid: lus25-MDT0000-lwp-OST0007_UUID }, ] nodemap.<span class="code-keyword">default</span>.fileset=nodemap.<span class="code-keyword">default</span>.id=0 nodemap.<span class="code-keyword">default</span>.squash_gid=99 nodemap.<span class="code-keyword">default</span>.squash_projid=99 nodemap.<span class="code-keyword">default</span>.squash_uid=99 nodemap.<span class="code-keyword">default</span>.trusted_nodemap=0   </pre> </div></div> <p> </p> <p>I have attached the full nodemap information form lus25 as an attachment. For comparision with another system we have.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus24-mds1 ~]# lctl get_param -R <span class="code-quote">'nodemap.<span class="code-keyword">default</span>'</span> nodemap.<span class="code-keyword">default</span>.admin_nodemap=0 nodemap.<span class="code-keyword">default</span>.audit_mode=1 nodemap.<span class="code-keyword">default</span>.exports=[ ] nodemap.<span class="code-keyword">default</span>.fileset=/<span class="code-keyword">null</span> nodemap.<span class="code-keyword">default</span>.id=0 nodemap.<span class="code-keyword">default</span>.squash_gid=65534 nodemap.<span class="code-keyword">default</span>.squash_uid=65534 nodemap.<span class="code-keyword">default</span>.trusted_nodemap=0   </pre> </div></div> <p> </p> <p>I am wondering if the issue is not with writes being poorly translated via nodemaps, but that the default nodemap being damaged somehow and therefore it appears to have all files owned by user 99 as I am accessing via the default nodemap which has uid squash turned on.</p> <p>Indeed, for lus25 you have admin and trusted properties to 0, so all accesses from nodes considered as part of this 'default' nodemap will be squashed to the ID 99 that you defined.</p> <p>I think the first thing to do would be the cleanup the situation regarding <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>, so that we work from a sane base.</p> <ul class="alternate" type="square"> <li>first problem is that your server side Lustre version 2.12.6_ddn66 is missing an important fix to <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>. This is explained in <a href="https://jira.whamcloud.com/browse/LU-15661" title="Nodemap possibly broken with upgrade" class="issue-link" data-issue-key="LU-15661"><del>LU-15661</del></a>. The fix is included in EXA5.2.5 (rpm tag 2.12.8-ddn6), and it needs to be applied on both server and client sides;</li> <li>on your custom 2.14 clients, you need to apply the 3 patches attached to this ticket, in this order: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/44850/44850_LU-14797-sec-add-projid-to-nodemap_b2_14_CLIENTONLY.patch" title="LU-14797-sec-add-projid-to-nodemap_b2_14_CLIENTONLY.patch attached to LU-16054">LU-14797-sec-add-projid-to-nodemap_b2_14_CLIENTONLY.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/44851/44851_LU-14797-nodemap-map-project-id_b2_14_CLIENTONLY.patch" title="LU-14797-nodemap-map-project-id_b2_14_CLIENTONLY.patch attached to LU-16054">LU-14797-nodemap-map-project-id_b2_14_CLIENTONLY.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/44852/44852_LU-15661-nodemap-fix-map-mode-value-for-both_b2_14_CLIENTONLY.patch" title="LU-15661-nodemap-fix-map-mode-value-for-both_b2_14_CLIENTONLY.patch attached to LU-16054">LU-15661-nodemap-fix-map-mode-value-for-both_b2_14_CLIENTONLY.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>. Please note theses patches are for 2.14 <b>client side</b> only, server part has been expunged from them.</li> </ul> <p>Do all clients including those using the default nodemap need those patches ?</p> <p>Are the patches in community release 2.15.0 ?</p> <blockquote> <p>Do all clients including those using the default nodemap need those patches ?</p></blockquote> <p>Yes, not matter which nodemap they are in, they can hit the problem.</p> <blockquote> <p>Are the patches in community release 2.15.0 ?</p></blockquote> <p>The patches are included in 2.15.0 (as well as EXA 5.2.5 and EXA 6.1).</p> <p>comparing lus24 which  works how we would expect with no mapping and lus25. I can see the squashed uid and gid are different  but the admin  trusted are the same ?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus24-mds1 ~]# lctl nodemap_test_nid 10.10.10.1@tcp0 Native</pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus25-mds1 exports]# lctl nodemap_test_nid 10.10.10.1@tcp <span class="code-keyword">default</span></pre> </div></div> <p>I think this an important difference.</p> <p>I can see no mapping for node 10.10.10.1@tcp in your definitions of lus25. Is that intended to have a different behavior than with lus24, which maps it to the 'Native' nodemap?</p> <p>What I am saying is that the two systems are different and the behaviour that is useful is the one lus24has.</p> <p> </p> <p>( How do we say all@tcp0 is native )</p> <blockquote> <p>I think the first thing to do would be the cleanup the situation regarding <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>, so that we work from a sane base.</p> <ul> <li>first problem is that your server side Lustre version 2.12.6_ddn66 is missing an important fix to <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>. This is explained in <a href="https://jira.whamcloud.com/browse/LU-15661" title="Nodemap possibly broken with upgrade" class="issue-link" data-issue-key="LU-15661"><del>LU-15661</del></a>. The fix is included in EXA5.2.5 (rpm tag 2.12.8-ddn6), and it needs to be applied on both server and client sides;</li> <li>on your custom 2.14 clients, you need to apply the 3 patches attached to this ticket, in this order: <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>-sec-add-projid-to-nodemap_b2_14_CLIENTONLY.patch <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a>-nodemap-map-project-id_b2_14_CLIENTONLY.patch <a href="https://jira.whamcloud.com/browse/LU-15661" title="Nodemap possibly broken with upgrade" class="issue-link" data-issue-key="LU-15661"><del>LU-15661</del></a>-nodemap-fix-map-mode-value-for-both_b2_14_CLIENTONLY.patch. Please note theses patches are for 2.14 client side only, server part has been expunged from them.</li> </ul> </blockquote> <p>The client part is actually optional to have <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a> fixed. This is nodemap related, and nodemap is not exported to client. So you can stick with your custom 2.14 client, the essential part being to have 2.12.8-ddn6 rpms installed on your servers.</p> <blockquote><p>Do all clients including those using the default nodemap need those patches ?</p> <p>Yes, not matter which nodemap they are in, they can hit the problem.</p></blockquote> <p>What I meant is <a href="https://jira.whamcloud.com/browse/LU-14797" title="lfs setquota should be nodemap aware" class="issue-link" data-issue-key="LU-14797"><del>LU-14797</del></a> is a server side problem, so no matter the nodemap to which clients belong (incl. default), they will be hit by the problem as long as server side is not fixed.</p> <p>I have just ran into this again on a newly reinstalled system...</p> <p> </p> <p>Server is</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus22-mds1 secure-lustre]# cat /sys/fs/lustre/version  2.12.9_ddn8</pre> </div></div> <p>Client is</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ubuntu@lus2526-tcp15:~$ cat /sys/fs/lustre/version  2.15.3</pre> </div></div> <p>I see</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus22-mds1 secure-lustre]#  lctl nodemap_test_nid 10.10.10.1@tcp <span class="code-keyword">default</span></pre> </div></div> <p>James</p> <p>This is puzzling. I understand that you've opened a support ticket to track this. Sébastien will need to review the logs supplied there and get back to you</p> <p>Peter</p> <p>Support pointed out "</p> <p>He has spotted one issue with the nodemap onfiguration though. there is no privileged nodemap, as explained in the Lustre Operations Manual:<br/> <a href="https://doc.lustre.org/lustre_manual.xhtml#idm140715165297696" class="external-link" target="_blank" rel="nofollow noopener">https://doc.lustre.org/lustre_manual.xhtml#idm140715165297696</a></p> <p>For proper operations, the Lustre file system requires to have a privileged group that<br/> covers all Lustre server nodes. So the very first step when working with nodemaps is to<br/> create such a group with both properties admin and trusted set. It is recommended to<br/> give this group an explicit label such as “TrustedSystems” or some identifier that makes<br/> the association clear."</p> <p>Hopefully next time we make that mistake this comment will remind me.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lus22-mds1 secure-lustre]# lctl nodemap_add Native [root@lus22-mds1 secure-lustre]# lctl nodemap_add_range --name Native --range <span class="code-quote">"[0-255].[0-255].[0-255].[0-255]@tcp"</span> [root@lus22-mds1 secure-lustre]# lctl nodemap_modify --name Native --property admin --value 1 [root@lus22-mds1 secure-lustre]# lctl nodemap_modify --name Native --property trusted --value 1</pre> </div></div> <p>Thank you and sorry for the confusion.</p> Related Development Rank 1|i02vn3: Rank (Obsolete) 9223372036854775807 Severity