Details
-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
-
Lustre 2.14.0, Lustre 2.16.0
-
None
-
3
-
9223372036854775807
Description
A number of sites using containers with Lustre have requested the ability to dynamically configure nodemaps on a regular basis to handle environments where they are being created and removed on a e.g. a per-job basis to contain only the nodes in the job for a specific runtime environment.
Storing all of the nodemaps in the config llog with frequent updates would quickly consume all of the space in the config log, and be slow to process during mounting.
Instead, there should be some option to allow using "lctl set_param nodemap.NODEMAP.PARAM" or an option to "lctl nodemap" to directly create and manage nodemaps without using the config llog to distribute the parameters to each server. This would require some external orchestration to set the nodemap parameters consistently across servers (e.g. "clush -a lctl set_param ...").
It might be desirable to have these dynamic nodemaps inherit settings from an existing persistent nodemap that would otherwise catch their NID range. This would have a number of benefits:
- safety in case of server restart so that missing dynamic nodemaps do not result in clients in the default nodemap
- avoid the need to set every parameter for the new nodemap, only change the subdirectory and NIDs and possibly a few other parameters
- (optionally?) make the dynamic nodemaps "safe by default" in that they can only add further restrictions rather than weaken them (e.g. longer subdirectory path, only disable access permissions, only remove UID mappings from the parent). Not totally sure about this, but seems interesting.
For identifying at mount time which nodemap is used, this could be hierarchical. Find the top-level NID range as today and map it to a nodemap, and consider this the "parent" nodemap. Then check if there is a second-level NID range below the parent and re-scan for matching NIDs and add extra restrictions from the child nodemap. Possibly repeat (max 10 levels for safety?).
That avoids making the initial NID searching complex (broad NID ranges can be used for the whole cluster) and then become more finegrained only if needed.
