[LU-17431] dynamically configurable nodemap

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.14.0, Lustre 2.16.0
• Severity: 3
• Assignee: Sebastien Buisson
• Reporter: Andreas Dilger

Description

A number of sites using containers with Lustre have requested the ability to dynamically configure nodemaps on a regular basis to handle environments where they are being created and removed on a e.g. a per-job basis to contain only the nodes in the job for a specific runtime environment.

Storing all of the nodemaps in the config llog with frequent updates would quickly consume all of the space in the config log, and be slow to process during mounting.

Instead, there should be some option to allow using "lctl set_param nodemap.NODEMAP.PARAM" or an option to "lctl nodemap" to directly create and manage nodemaps without using the config llog to distribute the parameters to each server. This would require some external orchestration to set the nodemap parameters consistently across servers (e.g. "clush -a lctl set_param ...").

It might be desirable to have these dynamic nodemaps inherit settings from an existing persistent nodemap that would otherwise catch their NID range. This would have a number of benefits:

• safety in case of server restart so that missing dynamic nodemaps do not result in clients in the default nodemap
• avoid the need to set every parameter for the new nodemap, only change the subdirectory and NIDs and possibly a few other parameters
• (optionally?) make the dynamic nodemaps "safe by default" in that they can only add further restrictions rather than weaken them (e.g. longer subdirectory path, only disable access permissions, only remove UID mappings from the parent). Not totally sure about this, but seems interesting.

For identifying at mount time which nodemap is used, this could be hierarchical. Find the top-level NID range as today and map it to a nodemap, and consider this the "parent" nodemap. Then check if there is a second-level NID range below the parent and re-scan for matching NIDs and add extra restrictions from the child nodemap. Possibly repeat (max 10 levels for safety?).

That avoids making the initial NID searching complex (broad NID ranges can be used for the whole cluster) and then become more finegrained only if needed.

Issue Links

• is related to: LU-14657 nodemap asynchronous update creates inconsistent entries.
• is related to: LU-17217 Allow server to control/deny client connections
• is related to: LU-8229 SSK: allow multiple keys on a single nodemap
• is related to: LU-17410 Add per-nodemap capabilities mask
• is related to: LU-8946 lctl nodemap_add/del_range should accept non-contiguous ranges

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55539/
Subject: LU-17431 nodemap: introduce child_raise_privileges property
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 904a7f1e8396eaa4bb969ccf0214ac0591f8887e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55002/
Subject: LU-17431 tests: exercise dynamic nodemap on MDS
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7dd4b9d9b650a505fec2fffe559f836d89fc7c55

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54739/
Subject: LU-17431 nodemap: make dynamic nodemaps hierarchical
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e9278b8da16338db30ac34abff1f6bc3489352b0

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58088/
Subject: LU-17431 nodemap: modify all properties of a dynamic nm
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 976a14cd8c5c805b10294a921eec7a943d80bf47

[Gerrit Updater]

"Sebastien Buisson <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58088
Subject: LU-17431 nodemap: modify all properties of a dynamic nm
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8a5bcb51ac674da4f3335163c5cae148e5c702d9

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54742/
Subject: LU-17431 nodemap: introduce func to init nodemap properties
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 44565cbe9d3ba36d796fd4827dfcabe33382b40e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54735/
Subject: LU-17431 nodemap: introduce helper funcs for range handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 82bb43cc8fb2daa104585b5f1a1947c06f8fba2c

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54513/
Subject: LU-17431 nodemap: allow modifying properties of a dynamic nm
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: db818bd0401038cbfac65bfd2f6f7a006d7fa150

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54509/
Subject: LU-17431 nodemap: allow handling an id map for a dynamic nm
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fb3459589e98169b1af2858abc70fafdf59c313d

[Marc Vef]

sebastien, adilger,

I'm currently reviewing Sébastien's patches on this ticket, and I've two questions regarding the current patch series:

1. Is there any update on your previous comments on whether hierarchical nodemaps must be dynamic and in-memory only? It seems to me that hierarchical nodemaps can also be persistently stored on the nodemap IAM and do not need to be dynamic (temporary). The restriction would be that persistent hierarchical nodemaps must be created on the MGS while dynamic nodemaps can be temporarily created on the MDS/OSS.

On another note, if persistent hierarchical nodemaps would be beneficial, it could make sense to change the terminology a bit (as Sébastien hinted): Since hierarchical nodemaps do not need to be dynamic in this case, they do not need a special name or property as they "only" inherit the settings of the parent. "Dynamic" nodemaps could be called "temporary" nodemaps to be more descriptive and to indicate that they are not persistent.

2. There is also the question of how hierarchical nodemaps should work with multiple filesets. The proposed idea in #55614 is that a child fileset must be a subdirectory of the parent fileset, and thus doesn't weaken the namespace restrictions set by the parent nodemap. The same can be done for multiple filesets such that child filesets must always have an existing parent fileset in the parent nodemap and represent a subdir of it. Is this correct?