[LU-18929] Inconsistent Lustre parameters for new MDT/OST

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Hongchao Zhang

Description

The parameters had been set by "lctl set_param -P" in the params config llog will not be applied for the newly added MDTs or OSTs.

Issue Links

• is related to: LU-17250 Add new MDT to existing filesystem misses OST pools, nodemaps, and other config
• is related to: LU-17334 Client should handle dir/file/object created on newly added MDT/OST
• is related to: LU-17920 Add permanent TBF rules

Comments

[Etienne Aujames]

1. have the client re-process the entire params (and potentially FSNAME-client) config llog(s) when new OSTs or MDTs are added. However, this has a potential issue that there could be parameters that should not be applied multiple times (e.g. new PCC-RO or WBC rules that are "cumulative", per patch https://review.whamcloud.com/56218 ("LU-17920 mgs: handle compound permanent parameters").

I think this is the straight forward option. But we do not want to re-apply parameters for the existing OST/MDT (those parameters could have been modified locally on servers/clients by the admins or scripts). So the solution, for me, would be to re-read the remote llog (from cache or directly from the MGS) but only apply it on the newly added sys/ proc/ paths. This means to develop something with "lfs set_param" to specify the sys or proc paths (like an env var, e.g: LFS_PARAM_PATH)

4. "inherit" the modified parameter values from a pre-existing OSC/MDC device (that has finished processing the config llog). We couldn't just blindly copy struct client_obd from an existing OSC to a newly-allocated one, since it would contain a lot of pointers/lists/flags and other values that are not usable.

This is interesting, but all the OST/MDT are not necessarily the same. So we could apply different parameters value for different kinds of OST/MDT (e.g: flash OSTs vs rotational OSTs). And the clients/servers have no way of knowing which target to trust for the copy. 

[Andreas Dilger]

I think this is not a straight-forward issue to fix. The client keeps a record of the last config llog record that it has processed, so that if the config DLM lock is revoked it will only start processing new config records and not re-apply all of the previous config records each time.

This is the right thing to do under normal operation when there are new parameters being saved into the config llog (i.e. only apply the new changes), but when a new MDT or OST is added to the filesystem, any persistent parameters like osc.*.* or mdc.*.* in the params config llog will not be applied to the client until the next time it is mounted.

A somewhat similar situation happened with new MDTs not inhetiting

There are potentially a few different options to resolve this issue:

1. have the client re-process the entire params (and potentially FSNAME-client) config llog(s) when new OSTs or MDTs are added. However, this has a potential issue that there could be parameters that should not be applied multiple times (e.g. new PCC-RO or WBC rules that are "cumulative", per patch https://review.whamcloud.com/56218 ("LU-17920 mgs: handle compound permanent parameters").
2. have the client "save" the whole config llog locally, and use them immediately when initializing only the new OSC/MDC device(s). This is somewhat complicated because "lctl set_param" is called via UDEV on the client to actually apply the parameters, so there may be potentially complex rules to rewrite the parameter names from the old config records to apply only to the new OSC/MDC names.
3. store a "initial parameter" value in the kernel for each modified parameter that is updated when a parameter is changed, and is used to initialize the new OSC/MDC devices. This has the benefit that the parameters could be applied immediately during setup, before the OSC/MDC became active in a busy filesystem, to ensure that it has the correct parameter setting even before it is used. The drawback is that this would need to be a filesystem-specific tunable and not a global variable (to avoid clashes if two different filesystems are mounted on a client), and (depending on the implementation) it might need ongoing maintenance to add the "tunable initial parameter" support for every tunable parameter.
4. "inherit" the modified parameter values from a pre-existing OSC/MDC device (that has finished processing the config llog). We couldn't just blindly copy struct client_obd from an existing OSC to a newly-allocated one, since it would contain a lot of pointers/lists/flags and other values that are not usable.

If implemented properly, this could be generic enough to handle all OSC/MDC tunable values applied on the client, while not needing every parameter to implement support for this explicitly.

[Gerrit Updater]

"Hongchao Zhang <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58817
Subject: LU-18929 mgs: revoke params lock for target_reg
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dbe96ce78cd1fc7454c2836790efab374530d3ec