[LU-11077] Client-specific tunable parameter configuration

• Status: Open
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Ben Evans (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved

Description

With nodemaps, the MGS should be able to configure individual nodemaps to have different parameters, like JOBID_VAR if one cluster is scheduled by SLURM and one is scheduled by lfs.

Adding a "-N NODEMAP" argument to the "lctl set_param" command line would be a good interface, and store the parameters into an FSNAME-client:NODEMAP file on the MGS that is sent to clients that are part of the nodemap.

Issue Links

• is related to: LU-10360 use Imperative Recovery logs for clie...
• is related to: LU-15246 Add per device adaptive timeout param...
• is related to: LU-14639 confusion of lru_size=0 if lru-resize...
• is related to: LU-16931 add lctl mechanism (list_param?) to r...

Comments

[Andreas Dilger]

Yes, it would make sense to store nodemap-specific tunables in <fsname>-client:<nodemap> or params:<nodemap> on the MGS config log for those clients. This would be sent from the MGS to clients connecting as part of <nodemap>.

However, we'd need to decide whether the clients would first process all of the <fsname>-client config log first and params:<nodemap> afterward, or only <fsname>-client:<nodemap>, or if all of the <nodemap> records would go into the global log and be passed or filtered based on whether the client is part of <nodemap> or not?

[Ben Evans (Inactive)]

With most configurations, you move from the most general to the most specific.  So apply the settings for everyone, like OST pools, and each nodemap would get informaition specific to them (like jobID format and network tokens).  So more targetted settings override more global ones.

[Andreas Dilger]

I'm also wondering whether it makes sense to have a purely client-side configuration file like "/etc/lustre/client.params", suitable for use on unique clients where the configuration needs to be dramatically different (e.g. remote clients, large shared-memory machines, service nodes, etc), that contains lines in a form suitable for passing to "lctl set_param -F" (which uses YAML "param: value" format):

osc.*.max_pages_per_rpc: 16M
llite.testfs-*.max_cached_mb: 64G

and/or directly usable as input to "lctl set_param" (which uses "param=value" format) like:

osc.*.max_pages_per_rpc=16M
llite.testfs-*.max_cached_mb=64G

This configuration file would be processed by /sbin/mount.lustre after the client filesystem is mounted. This is preferable to trying to automatically run a script like "/etc/rc.d/rc.local" at a specific point in the boot sequence, but would be ineffective if the client is (re-)mounted after boot.

It isn't clear to me whether we would need to have "/etc/lustre/$fsname-client.params" or not, since "$fsname" could already be specified as part of the parameters if needed (e.g. "osc.$fsname-*.max_rpcs_in_flight=64" or similar).

[Andreas Dilger]

We may want to handle the nodemap-side configuration change as part of a larger config file makeover together with LU-10360, since that will essentially eliminate the need for the "<fsname>-client" config file describing all of the MDT/MDC and OST/OSC devices.

[James A Simmons]

You can do this with the udev hwdb in theory since all of lustre's tunables that are sysfs files are udev aware. https://www.freedesktop.org/software/systemd/man/hwdb.html

 

[Andreas Dilger]

James, I don't think hwdb looks like a good fit for this, since that appears to be mostly desktop-focussed, and frankly I'm not a big fan of systemd. It also has the issue that it doesn't appear that it would work for parameters in debugfs or procfs.

[James A Simmons]

Our admins were not excited by the idea of hwdb since it meant to them managing another file with puppet. Admins, at least at ORNL, don't like configuration type files. They want things that are scriptable. For example the LUTF work is being looked to for this reason. Our admins will be able to manage and monitor LNet with python.

[Andreas Dilger]

There is nothing to prevent running "lctl set_param" from a script to set client-specific parameters today.

[Andreas Dilger]

One potential issue with having a separate params:<nodemap> file is that the client would have to explicitly request this configuration file from the MGS at mount time, but the client itself does not even know it is part of <nodemap>. It seems impractical to "append" the params:<nodemap> records to the end of params, since this would mess up config llog processing on the client if some other parameter was added to params at runtime, since llog files are sequential and the client tracks its last-processed record index locally so that it will process only new records on each update.

It seems reasonable for clients to request some "extra" config file like params:nodemap (i.e. literally "nodemap") or similar, and it is up to the MGS to do the mapping. For many clients, that would be nodemap=default, so possibly params:default if it exists (or maybe that is just the regular params file and is ignored), and for clients in nodemap=XXX they would get params:XXX if it exists.

For compatibility, it might also make sense that the nodemap-specific params:XXX file is supplied to the client instead of params if it exists, and this replaces the regular params file. That might make a bit more work for the admin since parameters would need to be copied over to each of the params:XXX files, but it would allow for disjoint parameters settings (i.e. not everything in params is inherited by clients into params:XXX).