https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16749 Lustre <p>We discovered that a lustre server can have entries in its configuration log which can effect other filesystems. eg</p> <p>lctl set_param -P llite.*.max_cached_mb=2048</p> <p>When a filesystem is mounted which has this in the filesystems configuration log, the client will apply the option to all mounted filesystems.</p> <p>I believe that the client should only apply a configuration option from the log to elements of that filesystem</p> <p>I can imagine a site which has multiple vendors and for example a vendor putting options in that casued issues for their competitor.</p> <p> </p> LU-16749

apply a filter to the configuration log.

Bug Minor Open Unresolved WC Triage James Beal Tue, 18 Apr 2023 13:57:34 +0000 Thu, 2 Nov 2023 00:48:18 +0000 Lustre 2.12.9 0 5 <p>A filter exist already. The correct command to use is "<tt>lctl set_param lllite.my_filesystem-&#42;.max_cached_mb=2048</tt>". Just using '<tt>&#42;</tt>' means apply to everything.</p> <p>This has definitely been a problem in the past, especially if mounting two filesystems with different timeouts, since there <b>is</b> only a single global value for the timeout on a client node. </p> <p>Patches in <a href="https://jira.whamcloud.com/browse/LU-15246" title="Add per device adaptive timeout parameters" class="issue-link" data-issue-key="LU-15246"><del>LU-15246</del></a> and <a href="https://jira.whamcloud.com/browse/LU-9912" title="fix multiple client mounts with different server timeouts" class="issue-link" data-issue-key="LU-9912">LU-9912</a> are fixing the issue of having only a single global timeout, at_min, etc. tunable per client, by adding per-OBD timeout values. Review and testing of those patches would be helpful. I've definitely thought about how <tt>lctl</tt> or the kernel could map "{{timeout}" to the per-device parameter automatically, but I haven't come up with a good solution yet. </p> <p>Since '<tt>&#42;</tt>' is affecting all filesystems, this is a bit tricky to fix, because a single MGS may be serving multiple local filesystems, so it isn't possible to automatically map '<tt>&#42;</tt>' in <tt>lctl</tt> to contain an fsname in the MGS config. It might be possible to constrain this to a specific filesystem during mount, since the other filesystem is also going to re-apply the settings when it mounts. However, not all '<tt>&#42;</tt>' uses are for the device name (though usually that is true), so this may be more complicated than expected.</p> <p>James B., getting some specific details of which parameters you had problems with would help to understand what needs to be fixed. As James S. wrote, it is already <em>possible</em> to use <tt>fsname-&#42;</tt> instead of just <tt>&#42;</tt> for many cases, though this requires more familiarity with Lustre, and probably updates to the Lustre manual and man pages to emphasize the correct usage.</p> <p>James S, I understand that you can correctly set parameters, "<tt>lctl set_param lllite.my_filesystem-*.max_cached_mb=2048</tt>" is what bit us in particular. It was set in error on a new filesystem during debuging another issue. When the filesystem was put in to production then performance on all filesystems plummetted in a hard to find way. What I am asking for is that a lustre client only applies settings fetched from a remote MGS to the filesystem that that client is mounting at that time.</p> <p> </p> <p>To put this in context, we put the new fileserver in to production on the 27th of Feburary. We then had serval weeks were we were getting tickets for bad performance and the filesystems were being hit harder than normal but when we looked at a filesystem we got good performance for the level of load that we could see on the filesystem. By the 6th of March I was convinced that this was a issue and started investigating. And we found the issue by the 10th, and by the 14th we had returned performance to acceptable levels.</p> <p> </p> <p>As an aside, and I suspect this is more a specific vendor comment rather than a whamcloud one: There seems to be a practice of setting configuration in the servers where the clients are in a better position to know the best values, eg a 2TB machine will have a different setting for <tt>max_cached_mb than a machine with 64GB..</tt></p> <p> </p> <p>I am also interested in why there is a <tt>max_cached_mb setting at all, why should there be a limit to the amount of ram used by the cache ( But I feel that would be a new ticket, if warrents discussion ).</tt></p> <p> </p> <p>In particular I can imagine a black hat fs that has a config for a competitors system reducing their performance however I suspect human error as we saw is much more likely.</p> <p>Definitely the <tt>max_cached_mb</tt> parameter is a client setting. The fact that it is set on the server is purely for convenience so that every client mounting the filesystem does not have to update the configuration. </p> <p>As for why this parameter exists, it is to allow admins to limit the amount of RAM used by the filesystem to avoid impacting application memory. Some apps ate tuned for a specific amount of RAM, and the admins/users don't want Lustre to interfere with that.</p> <p>Allowing a value like "20%" or "80%" would allow this to be usable on heterogeneous clients and probably would not be hard to implement. It used to be that most nodes in a cluster were the same, and a few would get a client-side setting, but it makes sense to handle this more flexibly. </p> <p>As for limiting parameters to a single filesystem mount, that is definitely something that I'm interested to have fixed, it just hasn't happened yet. </p> <p>Andreas. Can we use this issue for limiting parameters to a single filesystem mount.</p> <p>I would like to see the percentage accepted as a value for <tt>max_cached_mb</tt> however I don't understand why the parameter needs to exist, the nearest equiverlent I know of for vfs cache is <tt>vfs_cache_pressure</tt> It seems to me that the kernel should "just" evict pages if a user space program wants the pages ( In a tightly coupled mpi program I can see this could be an issue over multiple nodes, but the use case for genomics is a random soup of work loads all decoupled, so I feel that we should have the setting as something like 95% ). Shall I make a new issue for the percentage ?</p> <p>Yes, a separate issue for <tt>max_cached_mb=N%</tt> would be good.</p> <p>I can assure you that this tunable to limit Lustre cache size is a feature that some sites want in HPC environments. If the cache limit is high then page cache management defers to the VM under pressure, but otherwise it can be difficult for the VM to manage pages because it can take tens of seconds for dirty data to be written when the server is busy. </p> <p>The patch <a href="https://review.whamcloud.com/51952" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/51952</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-17030" title="allow setting max_cached_mb with %" class="issue-link" data-issue-key="LU-17030"><del>LU-17030</del></a> llite: allow setting max_cached_mb to a %</tt>" was landed for 2.16.0 to fix one part of the issue discussed here.</p> Related LU-9912 LU-8750 LU-15246 LU-17030 Development Epic/Theme client Rank 1|i03j87: Rank (Obsolete) 9223372036854775807 Severity