Minor Description
When using nodemaps, a squashed projid causes the projid to be out of sync on the MDT inode and OST objects. This can be reproduced with a client associated with a non-privileged nodemap that has the following fields set, most notably, no projid mapping and map_mode set to all:
nodemap.tenant1.admin_nodemap=1
nodemap.tenant1.audit_mode=1
nodemap.tenant1.deny_unknown=0
nodemap.tenant1.deny_mount=0
nodemap.tenant1.exports=[]
nodemap.tenant1.fileset=
nodemap.tenant1.forbid_encryption=0
nodemap.tenant1.id=3
nodemap.tenant1.idmap=
[
{ idtype: uid, client_id: 1003, fs_id: 11000 },
{ idtype: uid, client_id: 1004, fs_id: 11001 },
{ idtype: gid, client_id: 1003, fs_id: 11000 }
]
nodemap.tenant1.map_mode=all
nodemap.tenant1.offset=
{
start_uid: 0,
limit_uid: 0,
start_gid: 0,
limit_gid: 0,
start_projid: 0,
limit_projid: 0
}
nodemap.tenant1.ranges=
[
{ id: 5, start_nid: 192.155.155.50@tcp, end_nid: 192.155.155.50@tcp }
]
nodemap.tenant1.rbac=file_perms,dne_ops,quota_ops,byfid_ops,chlg_ops,fscrypt_admin,server_upcall,ignore_root_prjquota,hsm_ops
nodemap.tenant1.readonly_mount=0
nodemap.tenant1.sepol=
nodemap.tenant1.squash_gid=65535
nodemap.tenant1.squash_projid=65535
nodemap.tenant1.squash_uid=65535
nodemap.tenant1.trusted_nodemap=0
The PROJID 42 was set on the somedir directory from an admin=1 trusted=1 client via lfs project -s -p 42 /lustre/somedir. The issue can be reproduced by creating and writing to a new file from the non-privileged client: dd if=/dev/zero of=/lustre/somedir/somefile bs=1M count=32
After writing to a file, debugfs reports the PROJID 42 on the MDT inode and PROJID 65535 on the OST object(s). Note that looking at the PROJIDs on the privileged client via lfs project only presents the value from the MDT inode, while lfs quota looks at the value from the OST objects. It is therefore not clear to the administrator that somefile consumes space for PROJID 65535.
After some investigation, the MDT inode seems to use an incorrect value in this case. This is because on file creation, the MDS inherits the value from the parent directory and sets it for the new MDT inode (without considering the client's nodemap). And since the MDS is part of a trusted nodemap, it uses the fs_id (in nodemap mapping language), i.e., PROJID 42, while returning the client_id, i.e., PROJID 65535, to the client. As a result, the client only sees the client_id as the squashed PROJID and must send the squashed PROJID during OST_WRITE to an OST, which it uses to write to the OST object, causing the inconsistency. In summary, it seems that this PROJID mismatch is a result of the MDT not considering the client's nodemap (on MDT inode creation, inheriting the value from the parent inode) while the OST does (taking the PROJID from the client).
Note that this situation also occurs when no PROJID is set on the directory (or inheritance is disabled) in which case the MDT inode uses the PROJID 0 and the OST object uses the value 65535.