|
Niu
Could you please look into this one?
Thanks
Peter
|
|
With dlmtrace, it is clear that the two processes are attempting the same shrink over and over again:
|
|
How many locks are on this OSS?
$ lctl get_param ldlm.namespaces.*.lock_count
It looks like there are only a handful of locks that could be shrunk (9 or 10 in this log), but they are all busy. It seems like it is MM cache pressure that is continually calling this shrinker, about 3000 times per second based on the timestamps shown.
I wonder if there is some kind of logic mismatch between the various "shrinker" APIs in different kernels and the return value from our current shrinker? It could cause an overflow or underflow and result in this thread being stuck in a nearly endless loop.
|
|
Yes, it could be nice if we can get the total granted locks.
It looks like there are only a handful of locks that could be shrunk (9 or 10 in this log)
seems we can't get this conclusion from the log, since kernel always try to shrink SHRINK_BATCH (128) slab items each time, and our shrinker function will distribute this number to each OST by: cancel_for_this_ost = 1 + locks_on_this_ost * 128 / total_locks.
I'm afraid there are lots of granted locks, that's why kernel called shrinker many times (kernel get totoal locks first, then call shrinker to shrink 128 each time, until the number less than 128).
The stack trace shows the shrinker is called from zone_claim():
while (shrink_slab(sc.nr_scanned, gfp_mask, order) &&
zone_page_state(zone, NR_SLAB) > limit)
;
Which means if any slab object is shrunk, the shrinker could probably be called repeatedly. So, looks kernel really relies on shrinker returning the exact shrunk count, however, look at the ldlm_pools_shrink(), the return value seems unpredictable because it just account total remain locks from 'nr' OSTs, but not necessarily from each OST. (imagine that there are multiple threads are changing the namespace list, shrinker callers or ldlm poool recalc thread)
I'm wondering our shrinker should just return -1 to kernel, since our ldlm srv pool shrinker just changes SLR, but never reclaim anything directly.
Another problem is the congestion on ldlm_namespace_lock, which could make things worse.
|
|
There were 176207 total, more or less evenly distributed among 15 namespaces. Here are the specific numbers when I retrieved them yesterday:
11502, 11748, 11411, 11384, 11239, 11745, 12448, 11489, 12016, 11864, 11378, 12215, 12003, 12018, 11747
|
|
Attached a systemtap script and its output. Shows the inputs and outputs of the ldlm_pools_shrink() and shrink_slab() functions when those functions return. Time is in microseconds.
|
|
So I suspect that Niu Yawei is correct about were we are hung up. Although in our kernel the loop in __zone_reclaim() looks like this:
slab_reclaimable = zone_page_state(zone, NR_SLAB_RECLAIMABLE);
if (slab_reclaimable > zone->min_slab_pages) {
[cut]
while (shrink_slab(sc.nr_scanned, gfp_mask, order) &&
zone_page_state(zone, NR_SLAB_RECLAIMABLE) >
slab_reclaimable - nr_pages)
;
|
|
According to systemtap, shrink_slab was called with order=0x1. And in __zone_reclaim(), nr_pages is set to "1 << order", so nr_pages is currently 2.
systemtap also shows shink_slab returning rather large numbers (0x4fa9cf, 0x529d2d, 0x53ac3f, etc.). And I don't see any sign that the granted locks are reducing in number. They've grown to over 400,000 since yesterday, and seem to be hovering there. Other OSS nodes in this filesystem have similar lock totals.
|
|
I aggree that returning -1 from our shrinker will probably avoid this problem, because eventually shrink_slab() will be able to return 0 and jump out of the while() loop.
But I would like to understand better what is going on here. The use of the lock grant number in the lustre shrinker seems pretty far divorced from where the actual slab allocations happen. Why is it then using the grant number?
Conversely, why are we failing to ever reduce the zone_page_state()?
|
|
Thank you for the nice data and analysis, Chris.
One thing confused me is that why shrink_slab() return such large value? The shrink_slab() in my kernel looks like:
while (total_scan >= SHRINK_BATCH) {
long this_scan = SHRINK_BATCH;
int shrink_ret;
int nr_before;
nr_before = (*shrinker->shrinker)(0, gfp_mask);
shrink_ret = (*shrinker->shrinker)(this_scan, gfp_mask);
if (shrink_ret == -1)
break;
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
total_scan -= this_scan;
cond_resched();
}
[cut]
return ret
Since the number of granted locks didn't change much, the 'ret' should be a relative small value. Could you check your shrink_slab() or could you tell me what's your kernel version? Thanks.
|
|
What does "SLV" stand for?
ldlm_srv_pool_shrink really appears to make no attempt to actually reduce slab usage. At least it is honest and returns 0. (and has a comment to that effect). It seems like it should at least signal some other thread to asynchronously attempt to shrink the cache. Maybe I am reading the code wrong (I don't really have a clear idea of everything that is going on), but the comments seem to imply that we are just hoping that clients will be nicer because of the SLV change, and eventually, some day, they will use fewer locks.
What am I missing?
So it seems to me like ldlm_pools_shrink() is doing the Wrong Thing by ignoring the return code and looking at ldlm_pool_granted().
Is ldlm_pools_shrink() EVER right? Would it be right for LDLM_NAMESPACE_CLIENT?
|
Conversely, why are we failing to ever reduce the zone_page_state()?
I suspect that each time shrinker doesn't reduce the SLV much (only 9 ~ 10), and since we have a thread which recacluates the SLV every second, then SLV might be kept around a certain high value, so lock cancel wasn't triggered on client.
I think we need to drop the SLV to small value, then client could be aware of this change and start lock cancel.
|
|
Niu Yawei, the kernel is RHEL6.2-based, 2.6.32-220.4.1.1chaos. shrink_slab() is pretty much the same in the section that you quote, except for a new "struct shrinker *" that is passed as the first argument tot he (*shrinker->shrinker) callback.
|
One thing confused me is that why shrink_slab() return such large value?
I am a bit confused there too. See the new attached file "diff_between_before_and_after.txt".
Each number in that file is essentially the result of doing "nr_before - shrink_ret" by using the numbers for a single thread that got from a run of my systemtap script.
shrink_slab() probably has a large "total_scan" number, and needs to loop many times. shrink_ret is always a large positive number.
The difference, as you can see in the file, is wildly variable. But it is rarely 0, and when it is, it is not 0 repeatedly.
For calculating "ret", shrink_slab only considers the positive differences, and only increases "ret".
So I guess that explains it.
Which just brings me back to my suspicion that having ldlm_pools_shring return the summed value from ldlm_pool_granted() does not make sense.
|
|
SLV stands for Server Lock Volume. There is large piece of comment in the head of ldlm_pool.c to explain it, in general, SLV is used to inform client how many locks it should cache, if server decreases SLV, client will release cached locks when it's aware of the SLV change.
So the shrinker on server doesn't reclaim any memory directly, it just decrease the SLV, then client will cancel the cached locks to alleviate the server's memory pressure. I think this mechanism should work, however, we must make sure that:
- server shrinker should decrease the SLV significantly, and the SLV should not be increased back by the recalc thread immediately;
- client can response the decreased SLV quickly;
- the server shriner should be only called once, instead of many times like current implementation;
|
|
Hi, Chris
Yes, the "total_scan" is about 2 * granted_locks, and according to the granted_locks and return value of shrink_slab(), the average diff is ~770, it's reasonable.
I made a patch which returning -1 for the server shrinker and decreasing SLV more in the shrinker. Could you take a try on it once it pass the auto-test? Thanks.
http://review.whamcloud.com/#change,2184
|
|
I tested returning -1 using this simple systemtap script, and as expected it got the stuck threads out of their loop. I'll try the patch on our test system soon.
|
|
Niu, I don't understand why you want to do this in the patch:
nr = cfs_atomic_read(&pl->pl_granted);
Well, I suspect that you are trying to just use a larger number for the reduction that happens here:
pl->pl_server_lock_volume = pl->pl_server_lock_volume - nr;
But since pl_server_lock_volume is a essentially locks*time (and maybe other stuff that I don't completely understand yet), subtracting a discrete lock number from SLV seems fundamentally flawed. If we want to reduce the lock count and free memory, I think we have to apply a scaling factor.
Also, SLV may be just plain broken in 2.1. I see that server_lock_volume is 2073600000 for every OST on every OSS of one of our 2.1 clusters right now. Granted, I don't know too much about the details of SLV yet, but I would not have expected them all to have the exact same number within an OSS, let alone across all OSS.
|
|
Chris, I think you are right, seems subtracting nr_granted in server shrinker isn't enough, maybe we should try to divide the SLV by two (or make it even small). Could you check both the SLV & CLV in proc? I think we should at least decrease the SLV smaller CLV in the shrinker.
Could you check the other values for your 2.1 cluster, they are all in the /proc/fs/lustre/ldlm/namespaces/*/pool/state & stats. Thank you.
|
|
We only have 2.1 servers at this point. Attached is the output of teh state and stats files. CLV is 0, but I assume that is normal on a server.
|
|
Whoops, last file only had stats. This one (servers_pool_state_and_stats2.txt) has both.
|
|
Thank you, Chris. Looks the SLV of each OST is still equal to the initial value: SLV == L * 36000 (which means grant speed was almost same as the planned speed), and I'm afraid that CLV will be much less than SLV, since CLV of each lock is calculated as: CLV = seconds_of_lock_age * unused_locks_on_client.
To trigger lock cancel on client under server memory pressure, I think we have to drop the SLV to a much smaller value in the shrinker, for example: SLV = granted_locks * 100 (allow all granted locks on 1 client for 100 seconds?).
|
|
Hi, Chris, I update the patch in http://review.whamcloud.com/#change,2184
For ldlm server pool shrinker, we just use it to decrease SLV,
but never reclaim any memory directly, so it should always return
-1 to inform the kernel to break the shrink loop.
For ldlm client pool shrinker, we should just return remaining
unused locks as per the kernel's expectation.
The ldlm server pool shrinker should drop SLV to a small enough
value to trigger the lock cancellation from clients, we choose
(granted_locks * 100) in this patch.
Could you try the patchset two after it's verified by auto-test? Thanks.
|
|
Andreas's review in gerrit matches my own thinking.
I applied version 1 of your patch to our tree, since reducing SLV by nr is unlikely to have much effect, but returning -1 will help us avoid our immediate production problems.
But like Andreas said, we need to take a broader look into how SLV is calculated.
|
|
I agree with Andreas too. We need to reconsider if the current SLV calculation is proper.
Actually, I don't quite see why we want to control the grant speed at the very beginning, I think it's really hard to make it correct & efficient. To my understanding, the major purpose of LRU resize is to eliminate the fixed LRU size on client, and the only thing we need to care about is the limited memory on server and client, so we probably can use some simpler solution to control the total granted locks on server or unused locks on client (it should be easier than the grant flow control). That's just my initial thinking, and I'm not sure if miss anything important.
Anyway, it's good to hear that returnning -1 resloved your immediate problem, to improve the SLV calculation, I think we need more eyes and time-slot.
|
|
Niu, can you please update the patch with just the "-1" change. This bug (or a new one linked to this) should stay open until we have done a more complete investigation of the LDLM pool shrinker and SLV handling.
|
|
Integrated in  lustre-master » x86_64,server,el5,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,server,el5,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,el5,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,client,el5,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,server,el5,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,sles11,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,el5,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,server,el5,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,client,el6,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,client,el5,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,server,el6,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,el6,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,server,el6,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,server,el6,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » i686,client,el6,ofa #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,client,el6,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-master » x86_64,server,el6,inkernel #504
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Landed for 2.1.2 and 2.2
|
|
Please either reopen this, or open a new bug to track the SLV issue. Broken SLV is probably causing all kinds of problems. While we've supplied a work-around for the hang, the shrinker code is basically worthless at the moment.
|
|
Integrated in lustre-dev » x86_64,client,el5,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » i686,client,el6,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » i686,server,el5,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » x86_64,server,el6,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » i686,client,el5,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » x86_64,server,el5,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
Integrated in lustre-dev » x86_64,client,el6,inkernel #340
LU-1128 ldlm: return -1 for server pool shrinker (Revision 8775fe0a40706b6999aa737fba1c162ce08b8ade)
Result = SUCCESS
Oleg Drokin : 8775fe0a40706b6999aa737fba1c162ce08b8ade
Files :
|
|
opened LU-7266 to track the problems in LDLM pool.
|
Generated at Sat Feb 10 01:13:45 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
diff_between_before_and_after.txt
ldlm_pools_shrink2.stp