[LU-13335] add name lookup for project IDs

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: lug23dd lug24dd medium patch utils
• Assignee: Fred Dilger
• Reporter: Andreas Dilger

Description

It would be useful to allow names to be mapped to projid numbers using the /etc/projid file (projid.5), as is used by the XFS quota utility (xfs_quota.8):

usera:1
userc:3

There should be a new function struct ll_project *llapi_getprjname(const char *name) function that is called by lfs and lctl to read the /etc/projid file and lookup the name and return the projid, while struct ll_project *llapi_getprjid(__u32 prjid) would lookup the numeric project ID and return the name:

struct ll_project {
        __u32 lp_projid;
        char *lp_name;
};

Returning the struct instead of a single value (name or prjid) allows expanding this interface in the future (e.g. to list which users are allowed to use/change the group, passwords for the group, etc. as with getgrnam(), getgrgid(), getpwnam(), getpwuid()).

It isn't very clear what benefit the /etc/projects file (projects.5) provides, so I'm not yet sure whether we need to implement support for this or not.

This should be available for use by lfs project -p, lfs find -projid, lfs quota -p, lfs setquota -p.

Issue Links

• has to be done before: LU-18659 add new 'lfs projectadd' command to add project entries to the /etc/projid file
• has to be done before: LU-18638 print project name with 'lfs project'
• has to be done before: LU-18640 add option for 'lfs project' to do project name and id lookup only
• is related to: LU-14915 Add project flags to manage the project's directories and files
• is related to: LU-14968 add password for project files
• is related to: LU-17702 'lfs quota -u $USER' should print quota for all Lustre mountpoints
• is related to: LU-12056 tar doesn't support project id
• is related to: LU-18661 allow enable/disable projname->username fallback

Comments

[Andreas Dilger]

nathand, native LDAP projid support would require extensions to the glibc /etc/nsswitch and sssd functionality to allow a "project" database lookup, which is mostly out of scope for this feature. It would be useful to develop in the future, as proposed in my previous comment-322655:

It might potentially be useful to push a patch upstream for Linux /etc/nsswitch.conf to add the "project" as a valid database type. Since it is glibc, it may not be as obvious as one would hope (both technically and politically), but maybe just cargo-cult copying of e.g. "passwd" or "group" handling would be enough to start? Adding new getprojnam() and getprojid() calls would be needed also.

In the absence of sssd "project" database functionality, my previous proposal in comment-311464 was to overload the "passwd" database to do network-managed name->ID lookups, since many sites I'm familar with are using PROJID=UID for the user "home/" directories, and some mapping like "PROJID=(UID+1000000)" for "project/" and "PROJID=(UID+2000000)" for "scratch/" or similar:

I've also thought for a while that mapping username:uid to the same projid by default would be very convenient. For example, "lfs project -p adilger -r -s /lustre/home/adilger" with uid=1000 would fall back to use projid=1000 if there is no matching "adilger" entry in /etc/projid. That avoids the need to duplicate the entire /etc/passwd into /etc/projid for the common case where projid=uid, and potentially avoids the need to manage distributing /etc/projid to all of the client nodes since getpwnam() can query the configured network name database sssd (LDAP, AD, etc.) if so configured in /etc/nsswitch.conf.

It is reasonable that this fallback to use the passwd database to do UID->PROJID mapping should be configurable/disabled in /etc/projid, so that this functionality can be avoided if it is undesirable (e.g. there is no consistent UID->PROJID mapping, or network database access from the compute nodes is undesirable, or whatever is the reason as I don't like imposing arbitrary policy on the end user).

Alternately, this could be opt-in by specifying a "project name" like "-p u:adilger" or "-p g:adilger" to do a user or group database lookup, respectively, and use the resulting UID or GID for the PROJID. This puts a bit more burden on the admin to specify the database prefix correctly each time. I don't think it is valid to have a ':' in the username because /etc/passwd already uses this as a field separator, so that shouldn't cause any compatibility issues.

I don't mind having both mechanisms available, and it is up to the admin to use the one that they prefer, since this is only "lfs" internal complexity (and not very much at that), and the filesystem still only sees a single number as the end result.

[Nathan Dauchy]

It would be helpful to be able to extend this to be able to lookup Project ID in external sources such as LDAP.

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/46369/
Subject: LU-13335 ldiskfs: add projid to debug logs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6bceb0030d15b70097d75f8da2d373310fe2c658

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46369
Subject: LU-13335 ldiskfs: add projid to debug logs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8092bd00214ae19f87207914feeb97cab7faabbe

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46144
Subject: LU-13335 utils: allow projid lookup by name
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0ef6dd733f261aa2ce7788f4f0f6b29100f1979d

[Andreas Dilger]

It might potentially be useful to push a patch upstream for Linux /etc/nsswitch.conf to add the "project" as a valid database type. Since it is glibc, it may not be as obvious as one would hope (both technically and politically), but maybe just cargo-cult copying of e.g. "passwd" or "group" handling would be enough to start? Adding new getprojnam() and getprojid() calls would be needed also.

[Andreas Dilger]

There is also a more extensive /etc/project format defined by Solaris, that is apparently also handled by /etc/nsswitch.conf, so we should maintain compatibility with that for extensions beyond the basic projname:projid fields, essentially just ignoring any extra fields for now:

projname:projid:comment:user-list:group-list:attributes

The fields are defined as follows:

• projname - The name of the project. The name must be a string that consists of alphanumeric characters, underline (_) characters, hyphens (-), and periods (.). The period, which is reserved for projects with special meaning to the operating system, can only be used in the names of default projects for users. projname cannot contain colons (:) or newline characters.
• projid - The project's unique numerical ID (PROJID) within the system. The maximum value of the projid field is UID_MAX (2147483647).
• comment - A description of the project.
• user-list - A comma-separated list of users who are allowed in the project. Wildcards can be used in this field. An asterisk (*) allows all users to join the project. An exclamation point followed by an asterisk (!*) excludes all users from the project. An exclamation mark (!) followed by a user name excludes the specified user from the project.
• group-list - A comma-separated list of groups of users who are allowed in the project. Wildcards can be used in this field. An asterisk (*) allows all groups to join the project. An exclamation point followed by an asterisk (!*) excludes all groups from the project. An exclamation mark (!) followed by a group name excludes the specified group from the project.
• attributes - A semicolon-separated list of name-value pairs, such as resource controls (see Chapter 6, Resource Controls (Overview)). name is an arbitrary string that specifies the object-related attribute, and value is the optional value for that attribute.

      name[=value]
  

  In the name-value pair, names are restricted to letters, digits, underscores, and periods. A period is conventionally used as a separator between the categories and subcategories of the resource control (rctl). The first character of an attribute name must be a letter. The name is case sensitive.

Values can be structured by using commas and parentheses to establish precedence.

A semicolon is used to separate name-value pairs. A semicolon cannot be used in a value definition. A colon is used to separate project fields. A colon cannot be used in a value definition.

[Andreas Dilger]

I've also thought for a while that mapping username:uid to the same projid by default would be very convenient. For example, "lfs project -p adilger -r -s /lustre/home/adilger" with uid=1000 would fall back to projid=1000 if there is no matching "adilger" entry in /etc/projid. That avoids the need to duplicate the entire /etc/passwd into /etc/projid for the common case where projid=uid, and potentially avoids the need to manage distributing /etc/projid to all of the client nodes since getpwnam() can query the configured network name database (LDAP, AD, etc) if configured in /etc/nsswitch.conf.

[Andreas Dilger]

mrb, we had briefly discussed at LUG that you might work on this improvement. Is that still on your radar?

[Andreas Dilger]

It looks like there is already a stub lfs.c::name2projid() function used in most of the relevant code that needs to be filled in to convert the name to the numeric projid. That would need to call llapi_getprjid() for the name lookup.

The one place this is not done correctly is in lfs_project(), where it only calls str2quotaid(), unlike lfs_find(), lfs_setquota() and lfs_quota() that call the respective name2*id() function first, and only fall back to str2quotaid() if they return an error. (As an aside, str2quotaid() should rename the internal "projid_tmp" variable to "tmpid" since function this is used for parsing and verifying all kinds of numeric IDs).

[Andreas Dilger]

We don't manage username->uid or groupname->gid mapping in Lustre, but rather depend on external services for mapping. The XFS filesystem is using "/etc/projid" for projname->projid mapping, so it makes sense to be compatible with this.

Once the basic projname->projid mapping is handled, then we could potentially expand this to manage the configuration in Lustre, or via NIS or AD or whatever is in use at the site (as is done with getgrnam(), getgrgid(), ...), but that is a lot more work.