[LU-18835] 'chown' in nodemap sets UID/GID to '65534' on OST objects

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.17.0
• Severity: 3
• Assignee: Marc Vef
• Reporter: Marc Vef

Description

This is a situation that works correctly when no nodemap is used. The following describes how to reproduce this issue on a freshly-formatted Lustre file system.

The nodemap is set as follows for a single client, mapping two users that are part of the same group. The squashing uid is slightly modified (it becomes clear why later).

Nodemap config:

nodemap.draze.admin_nodemap=1
nodemap.draze.audit_mode=1
nodemap.draze.deny_unknown=0
nodemap.draze.deny_mount=0
nodemap.draze.exports=
[
 { nid: 192.155.155.50@tcp, uuid: 247b89e5-e7c5-46d9-8707-e0304329f54f }
]
nodemap.draze.fileset=
nodemap.draze.forbid_encryption=0
nodemap.draze.id=2
nodemap.draze.idmap=
[
 { idtype: uid, client_id: 1003, fs_id: 11000 },
 { idtype: uid, client_id: 1004, fs_id: 11001 },
 { idtype: gid, client_id: 1003, fs_id: 11000 }
]
nodemap.draze.map_mode=all
nodemap.draze.offset=
{
 start_uid: 0,
 limit_uid: 0,
 start_gid: 0,
 limit_gid: 0,
 start_projid: 0,
 limit_projid: 0
}
nodemap.draze.ranges=
[
 { id: 2, start_nid: 192.155.155.50@tcp, end_nid: 192.155.155.50@tcp }
]
nodemap.draze.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,hsm_ops
nodemap.draze.readonly_mount=0
nodemap.draze.sepol=
nodemap.draze.squash_gid=65534
nodemap.draze.squash_projid=65534
nodemap.draze.squash_uid=65535
nodemap.draze.trusted_nodemap=0

Next, Lustre is mounted on this tenant. The root dir is slightly modified.

Namespace setup:

# allow user 1003 to write to root
root@lu-draze-client ~ $ chown 1003:1003 /exafs

# Write to a file as a user:
draze-usr@lu-draze-client ~ $ dd if=/dev/zero of=/exafs/file bs=1M count=32
draze-usr@lu-draze-client ~ $ ls -ln /exafs
total 32768
-rw-rw-r--. 1 1003 1003 33554432 Mar 19 18:08 file

MDT object tagged UID/GID:

root@lu-ahulane-mds ~ $ debugfs -c -R "stat ROOT/file" ~/lustre_dev/lustre_mdt1
debugfs 1.47.0-wc1 (28-Apr-2023)
Inode: 1048335   Type: regular    Mode:  0664   Flags: 0x0
Generation: 2080448777    Version: 0x00000001:00000002
User: 11000   Group: 11000   Project:     0   Size: 0
..

OST object tagged UID/GID:

root@lu-evie-oss ~ $ debugfs -c -R "stat O/2c0000400/d$((2 % 32))/2" ~/lustre_dev/lustre_ost3
debugfs 1.47.0-wc1 (28-Apr-2023)
Inode: 103   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 3885368229    Version: 0x00000001:0000000b
User: 11000   Group: 11000   Project: 65534   Size: 33554432
..

chown file to 1004:1003:

root@lu-draze-client ~ $ chown 1004:1003 /exafs/file

# verify ids:
draze-usr@lu-draze-client ~ $ ls -ln /exafs
total 32768
-rw-rw-r--. 1 1004 1003 33554432 Mar 19 18:08 file

MDT object tagged UID/GID after chown (correct):

root@lu-ahulane-mds ~ $ debugfs -c -R "stat ROOT/file" ~/lustre_dev/lustre_mdt1
debugfs 1.47.0-wc1 (28-Apr-2023)
Inode: 1048335   Type: regular    Mode:  0664   Flags: 0x0
Generation: 2080448777    Version: 0x00000001:00000004
User: 11001   Group: 11000   Project:     0   Size: 0

OST object tagged UID/GID after chown (wrong, should be equivalent to MDT object):

root@lu-evie-oss ~ $ debugfs -c -R "stat O/2c0000400/d$((2 % 32))/2" ~/lustre_dev/lustre_ost3
debugfs 1.47.0-wc1 (28-Apr-2023)
Inode: 103   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 3885368229    Version: 0x00000001:0000000b
User: 65534   Group: 65534   Project: 65534   Size: 33554432

Note that the "User" and "Group" IDs changed to 65534 for the OST object while the IDs are set correctly on the MDT object.

I've also checked this for multiple stripes, in which case 65534 was set on all stripe OST objects. While this issue is related to nodemaps, it is unrelated to what is set as the squashed ids on the nodemap. The uid was set to 65535 earlier to verify whether it is used to tag the object UID, which is seemingly not the case. The UIDs 11000 and 11001 as well as the GID 11000 were set on both MDS and OSS. Writing to the file does not change "User" and "Group" of the OST object.

When no nodemap is used, the "User" and "Group" IDs are set correctly on the OST object.

Issue Links

• is related to: LU-18857 tunable parameter to allow/deny new MDT/OST registration with MGS
• is related to: LU-4966 MGS target registration should use proper UUID
• is related to: LU-18856 MGS nodemap verification

Comments

[Andreas Dilger]

I think that this is an artifact of the MDS node not being configured in the "trusted" nodemap, and should not happen on a properly configured system where the MDT and OST are not mediated by a nodemap in their communication (ID mapping/offset, local_admin, root squash, RBAC, etc).

This brings me back to my earlier thought that in addition to the "default" nodemap, it would make sense to always create a "trusted" nodemap where all of the server NIDs in the MGS FSNAME-OSTxxxx/FSNAME-MDTxxxx config llog are added so that they are sure to be working properly.

[Gerrit Updater]

"Marc Vef <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58520
Subject: LU-18835 target: Skip id mapping for OST_SETATTR from MDS
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9e947a68940661a3fcb6d36f30eb3c2b00773fb0

[Sebastien Buisson]

At least 65534 is the value defined for squash_uid on your nodemap.

[Marc Vef]

No, not yet. This is only an observation so far, and I'll have to look further into it. I guess 65534, which is used to tag the OST objects, is some kind of default value somewhere.

[Sebastien Buisson]

That's seems like a real issue to me. So the ids are squashed on OST side, have you been able to identify or trace what is the 'input' id being squashed?

[Marc Vef]

sebastien, does this ring a bell or is this a legitimate issue?