[LU-16931] add lctl mechanism (list_param?) to report changes to tunable parameters

Related — Sat, 20 Jan 2024 05:41:14 +0000

It would be useful to have a generic mechanism that reports when tunable parameters are modified. For example with "lctl list_param -FR '*'" it will print whether the entry is a directory and if the parameter is writable with trailing "/=":

$ lctl list_param -FR '*'
:
jobid_name=
jobid_this_session=
jobid_var=
lbug_on_eviction=
ldlm/
ldlm.cancel_unused_locks_before_replay=
ldlm.namespaces/
ldlm.namespaces.MGC192.168.20.1@tcp/
ldlm.namespaces.MGC192.168.20.1@tcp.dirty_age_limit=
ldlm.namespaces.MGC192.168.20.1@tcp.early_lock_cancel=
ldlm.namespaces.MGC192.168.20.1@tcp.lock_count
:

Firstly, being able to list only writeable parameters with "lctl list_param -w PATTERN" would be useful for separating tunables from read-only parameters.

Secondly, and more importantly, being able to easily find/report parameters that were actually modified on the client or server for whatever reason (either from saved parameters on the MGS or manually with "lctl set_param") would make debugging system configuration issues much easier.

One limitation is that these parameters are exposed to userspace via /proc and /sys files, so just storing an internal flag for the modified parameter itself does not give us any way to report them directly to userspace.

One option would be to change the mtime of the parameter inode when it is modified? This might make it possible to distinguish those parameters that have a different mtime from the ctime. Currently they are always the same:

# find /{proc,sys}/fs/lustre /sys/kernel/debug/lustre -type f | grep lru_size | xargs ls -lc
-rw-r--r--. 1 root root 4096 Mar  9 11:33 /sys/fs/lustre/ldlm/namespaces/MGC192.168.20.1@tcp/lru_size
-rw-r--r--. 1 root root 4096 Apr 16 02:09 /sys/fs/lustre/ldlm/namespaces/myth-MDT0000-mdc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:31 /sys/fs/lustre/ldlm/namespaces/myth-OST0000-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Apr 16 02:09 /sys/fs/lustre/ldlm/namespaces/myth-OST0001-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:32 /sys/fs/lustre/ldlm/namespaces/myth-OST0002-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:32 /sys/fs/lustre/ldlm/namespaces/myth-OST0003-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:31 /sys/fs/lustre/ldlm/namespaces/myth-OST0004-osc-ffff979380fc1800/lru_size
# find /{proc,sys}/fs/lustre /sys/kernel/debug/lustre -type f | grep lru_size | xargs ls -lu
-rw-r--r--. 1 root root 4096 Mar  9 11:33 /sys/fs/lustre/ldlm/namespaces/MGC192.168.20.1@tcp/lru_size
-rw-r--r--. 1 root root 4096 Apr 16 02:09 /sys/fs/lustre/ldlm/namespaces/myth-MDT0000-mdc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:31 /sys/fs/lustre/ldlm/namespaces/myth-OST0000-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Apr 16 02:09 /sys/fs/lustre/ldlm/namespaces/myth-OST0001-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:32 /sys/fs/lustre/ldlm/namespaces/myth-OST0002-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:32 /sys/fs/lustre/ldlm/namespaces/myth-OST0003-osc-ffff979380fc1800/lru_size
-rw-r--r--. 1 root root 4096 Mar  9 11:31 /sys/fs/lustre/ldlm/namespaces/myth-OST0004-osc-ffff979380fc1800/lru_size

I believe the different timestamps in this case are because the MDT0000 and OST0001 devices were remounted after the filesystem was first mounted. However, I haven't done any investigation on whether the mtime/ctime can be set differently for sysfs/procfs inodes, but I suspect they use regular struct inode and could be modified appropriately.

Another option would be to list the modified parameter names into an internal table and have a new parameter file that prints the modified parameter names. This could potentially be tricky to implement, because it isn't always clear from the parameter handler itself what the full pathname to the modified parameter is. However, this has the advantage that the parameter could easily be dumped (e.g. "lctl get_param modified_params") and would itself be included in a "dump all parameters" output like "lctl get_param -R '*'".

In addition to dumping the names of the modified parameters, it would be possible to actually keep a log of the original/new parameters, and what time they were changed, etc. However, at that point we might just consider to log all parameter changes to /var/log/messages so that they are readily available without any additional effort. The main drawback of this is on large clusters with many clients, as this may produce a lot of log spam if 10000 clients all report "parameter max_dirty_mb changed from 2048 to 4096" for dozens of parameters, especially if the contents are large.

Whamcloud Community JIRA

[LU-16931] add lctl mechanism (list_param?) to report changes to tunable parameters