https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14517 Lustre <p>lru_max_age is set to 65min for ages. This is a looong time and that makes clients keep lots of LDLM locks in cache, and more data in cache. The only reason for that, that I'm aware of, is to help login nodes to avoid them requesting the same locks too often.</p> <p>I know a lot of sites tune this value to something much smaller. Compute nodes have a more agressive value usually, and the LRU cache is clear between jobs anyway.</p> <p>I think it would be valuable to change the default value to a much reasonable one to avoid most of users to decrease this value. I think going down to 5 min would be a good move.</p> <p>If login nodes has to re-enqueue some locks every 5 min I think it is not a problem at all.</p> <p> </p> <p>What do you think of that?</p> LU-14517

Decrease default lru_max_age value

Improvement Minor Open Unresolved WC Triage Aurelien Degremont Fri, 12 Mar 2021 17:05:30 +0000 Sun, 21 Jan 2024 19:14:59 +0000 0 4 <p>The reason this is done is that a common work case is to do a checkpoint every hour.</p> <p>Aurelien, I wouldn't be against this. Maybe 5 minutes is a bit too short, but 10 minutes is better? This is set with <tt>LDLM_DEFAULT_MAX_ALIVE</tt> value. </p> <p>See also <a href="https://jira.whamcloud.com/browse/LU-6402" title="reduce the value of LDLM_POOL_MAX_AGE" class="issue-link" data-issue-key="LU-6402">LU-6402</a>, which is the <tt>LDLM_POOL_MAX_AGE</tt> value, which is still at the <em>very</em> old 36000s/10h value, which controls how (badly) the dynamic LRU pressure on the client is keeping locks on the client. It makes sense to reduce this to at least 3600s, but probably also lower.</p> <p>Aurelien, note that <tt>lru_max_age</tt> should really only be a fallback upper limit for the dynamic LRU pool management. Unfortunately, the dynamic LRU code has not been working well for a long time (see <a href="https://jira.whamcloud.com/browse/LU-7266" title="Fix LDLM pool to make LRUR working properly" class="issue-link" data-issue-key="LU-7266">LU-7266</a>), and often users disable it by setting <tt>lru_max_age</tt> and <tt>lru_size=N</tt>.</p> <p>However, that is sub-optimal since it means some clients may have too many locks, while others too few, and setting too high a limit causes memory pressure on the servers and/or clients.</p> <p>What is really needed here is some investigation into the LDLM pool "Lock Volume" calculations to see why this is not working. The basic theory is that sum(age of locks) is a "volume" that the server distributes among clients, and the client can manage locks within that volume as it sees fit (many short-lived locks, few long-lived locks), and if the client lock volme is growing to exceed its assigned limit (due to aging of old locks and/or acquiring many new locks) then it should cancel the oldest unused locks to reduce the volume again. The client is really in the best position to judge which of its locks are most important, but as a workaround to memory pressure issues, <a href="https://jira.whamcloud.com/browse/LU-6529" title="Server side lock limits to avoid unnecessary memory exhaustion" class="issue-link" data-issue-key="LU-6529"><del>LU-6529</del></a> was implemented to give the server the ability to cancel locks more aggressively to avoid OOM.</p> <p>It <em>may</em> be that <tt>LDLM_POOL_MAX_AGE</tt> is just set much too high and/or the DLM server is allowing too much memory to be put toward locks (e.g. not considering multiple namespaces, or just assigning too large a fraction of RAM to LDLM vs. filesystem cache, etc), so the clients are not cancelling locks aggressively enough. There may also be issues w.r.t. hooking into the kernel slab cache shrinkers not working properly (this should reduce the lock volume on the server to force clients to cancel locks, and on the client to directly cancel locks).</p> <p>The other area that could benefit is replacing the strict LRU managing the locks on the client. For clients doing things like filesystem scanning, strict LRU is not a very good algorithm, since that flushes out "valuable" locks too quickly (e.g. parent directory locks) and doesn't drop "boring" locks (e.g. the use-once locks for the individual files). Using a better caching algorithm (e.g. LFRU, 2Q/SLRU, ARC) would go a long way to improving lock cache usage on the client. ARC is probably the best choice, since it would be possible to keep the FIDs in the "ghost" lists without actually caching the lock/pages, and in case some frequently-used lock had to be cancelled due to contention it doesn't immediately lose the "value" that had been built up for that lock.</p> <p>I can see the bad effects of a fixed size <tt>lru_size</tt>. That's why I'm not using it and this value makes more sense as a read-only one nowadays.</p> <p>However, I may have been misunderstanding the behavior of lru_max_age. I'm using it as a way to limit dynamically number of locks on client side, based on access pattern. I understood it was the time since the resource last access. FS scan can bring a lot of locks on client indeed, but every time a directory is accessed again I thought it was considered "young" again and not evicted from cache. I'm also using that to force client to flush dirty cache more aggressively.</p> <p>You are correct <tt>lru_max_age</tt> is based on the idle time of a lock, which is refreshed on lock usage. However, consider the competing needs of keeping an often used lock in cache vs. flushing many use-once locks from cache. If <tt>lru_max_age</tt> is high, it helps locks that may be used repeatedly, but not continuously, but if <tt>lru_max_age</tt> is high then the client may accumulate a large number of use-once locks before they age out. If <tt>lru_max_age</tt> is low to keep too many use-once locks out of cache locks reused many times may also be cancelled if there is any gap in their usage. Currently, there is no frequency counter on locks, so all locks that hit <tt>lru_max_age</tt> (or <tt>lru_size</tt> if set) will be cancelled regardless of how useful they are to the client. That is what <a href="https://jira.whamcloud.com/browse/LU-11509" title="LDLM: replace lock LRU with improved cache algorithm" class="issue-link" data-issue-key="LU-11509">LU-11509</a> is about - improving the algorithm for flushing locks from the cache instead of strict LRU.</p> <p>I was not against the advantages of replacing the LRU algorithm by something else like ARC, I think it makes sense.</p> <p>My point was only that: 65minutes as the default value is too big in my opinion, that is making clients keep unused locks for that long. If we reduce that to, let's say 10 minutes, the lock volume will be smaller, the lock callback traffic will be smaller (less lock conflicts, less eviction due to lock callbacks) at a price of a increased lock enqueue traffic, which I suspect will be small. I'm curious to know the p90 lock age on most clients.</p> <p> </p> Related LU-7266 LU-6402 LU-6529 LU-11509 Development Rank 1|i01pcf: Rank (Obsolete) 9223372036854775807