https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11077 Lustre <p>With nodemaps, the MGS should be able to configure individual nodemaps to have different parameters, like <tt>JOBID_VAR</tt> if one cluster is scheduled by SLURM and one is scheduled by lfs.</p> <p>Adding a "<tt>-N NODEMAP</tt>" argument to the "<tt>lctl set_param</tt>" command line would be a good interface, and store the parameters into an <tt>FSNAME-client:NODEMAP</tt> file on the MGS that is sent to clients that are part of the nodemap.</p> LU-11077

Client-specific tunable parameter configuration

Improvement Minor Open Unresolved WC Triage Ben Evans Fri, 8 Jun 2018 20:58:30 +0000 Thu, 25 Jan 2024 07:10:45 +0000 0 6 <p>Yes, it would make sense to store nodemap-specific tunables in <tt>&lt;fsname&gt;-client:&lt;nodemap&gt;</tt> or <tt>params:&lt;nodemap&gt;</tt> on the MGS config log for those clients. This would be sent from the MGS to clients connecting as part of <tt>&lt;nodemap&gt;</tt>.</p> <p>However, we'd need to decide whether the clients would first process all of the <tt>&lt;fsname&gt;-client</tt> config log first and <tt>params:&lt;nodemap&gt;</tt> afterward, or only <tt>&lt;fsname&gt;-client:&lt;nodemap&gt;</tt>, or if all of the <tt>&lt;nodemap&gt;</tt> records would go into the global log and be passed or filtered based on whether the client is part of <tt>&lt;nodemap&gt;</tt> or not?</p> <p>With most configurations, you move from the most general to the most specific.  So apply the settings for everyone, like OST pools, and each nodemap would get informaition specific to them (like jobID format and network tokens).  So more targetted settings override more global ones.</p> <p>I'm also wondering whether it makes sense to have a purely client-side configuration file like "<tt>/etc/lustre/client.params</tt>", suitable for use on unique clients where the configuration needs to be dramatically different (e.g. remote clients, large shared-memory machines, service nodes, etc), that contains lines in a form suitable for passing to "<tt>lctl set_param -F</tt>" (which uses YAML "<tt>param: value</tt>" format):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc.*.max_pages_per_rpc: 16M llite.testfs-*.max_cached_mb: 64G </pre> </div></div> <p>and/or directly usable as input to "<tt>lctl set_param</tt>" (which uses "<tt>param=value</tt>" format) like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc.*.max_pages_per_rpc=16M llite.testfs-*.max_cached_mb=64G </pre> </div></div> <p>This configuration file would be processed by <tt>/sbin/mount.lustre</tt> after the client filesystem is mounted. This is preferable to trying to automatically run a script like "<tt>/etc/rc.d/rc.local</tt>" at a specific point in the boot sequence, but would be ineffective if the client is (re-)mounted after boot.</p> <p>It isn't clear to me whether we would need to have "<tt>/etc/lustre/$fsname-client.params</tt>" or not, since "$<tt>fsname</tt>" could already be specified as part of the parameters if needed (e.g. "<tt>osc.$fsname-*.max_rpcs_in_flight=64</tt>" or similar).</p> <p>We may want to handle the nodemap-side configuration change as part of a larger config file makeover together with <a href="https://jira.whamcloud.com/browse/LU-10360" title="use Imperative Recovery logs for client-&gt;MDT/OST connections" class="issue-link" data-issue-key="LU-10360">LU-10360</a>, since that will essentially eliminate the need for the "<tt>&lt;fsname&gt;-client</tt>" config file describing all of the MDT/MDC and OST/OSC devices.</p> <p>You can do this with the udev hwdb in theory since all of lustre's tunables that are sysfs files are udev aware. <a href="https://www.freedesktop.org/software/systemd/man/hwdb.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.freedesktop.org/software/systemd/man/hwdb.html</a></p> <p> </p> <p>James, I don't think hwdb looks like a good fit for this, since that appears to be mostly desktop-focussed, and frankly I'm not a big fan of systemd. It also has the issue that it doesn't appear that it would work for parameters in debugfs or procfs. </p> <p>Our admins were not excited by the idea of hwdb since it meant to them managing another file with puppet. Admins, at least at ORNL, don't like configuration type files. They want things that are scriptable. For example the LUTF work is being looked to for this reason. Our admins will be able to manage and monitor LNet with python.</p> <p>There is nothing to prevent running "<tt>lctl set_param</tt>" from a script to set client-specific parameters today. </p> <p>One potential issue with having a separate <tt>params:&lt;nodemap&gt;</tt> file is that the client would have to explicitly request this configuration file from the MGS at mount time, but the client itself does not even know it is part of <tt>&lt;nodemap&gt;</tt>. It seems impractical to "append" the <tt>params:&lt;nodemap&gt;</tt> records to the end of <tt>params</tt>, since this would mess up config llog processing on the client if some other parameter was added to <tt>params</tt> at runtime, since llog files are sequential and the client tracks its last-processed record index locally so that it will process only new records on each update.</p> <p>It seems reasonable for clients to request some "extra" config file like <tt>params:nodemap</tt> (i.e. literally "<tt>nodemap</tt>") or similar, and it is up to the MGS to do the mapping. For many clients, that would be <tt>nodemap=default</tt>, so possibly <tt>params:default</tt> if it exists (or maybe that is just the regular <tt>params</tt> file and is ignored), and for clients in <tt>nodemap=XXX</tt> they would get <tt>params:XXX</tt> if it exists. </p> <p>For compatibility, it might also make sense that the nodemap-specific <tt>params:XXX</tt> file is supplied to the client <b>instead</b> of <tt>params</tt> if it exists, and this <em>replaces</em> the regular <tt>params</tt> file. That might make a bit more work for the admin since parameters would need to be copied over to each of the <tt>params:XXX</tt> files, but it would allow for disjoint parameters settings (i.e. not everything in <tt>params</tt> is inherited by clients into <tt>params:XXX</tt>).</p> Related LU-10360 LU-15246 LU-14639 LU-16931 Development Rank 1|hzzy73: Rank (Obsolete) 9223372036854775807