|
Can you please do the tip of b2_8 testign to see if it's also affected?
Also on master - if you can enable memory allocation tracing, reproduce, and then extract the debug log out of a crashdump so that we can see where are all of those allocations come from.
|
|
Frank or Cliff,
it might be useful to increase the debug buffer size and turn on trace logging (lctl set_param debug_mb=128 debug=+trace) and then let it continue to run for a minute or two and dump the debug log without having to wait for the crash to be hit. The size-1024 items are being allocated 10/sec so the cause will appear in any logs. Similarly, the size-1048576 allocations are done every minute so the cause should also be in any logs dumped. The size-262144 is only being allocated in chunks every 15 minutes, so it might be related to recovery if that is the frequency of failovers, so that might need to be timed to coordinate with a failover or similar, or the size of the log increased to 512 or 1024 MB.
Di, the memory is all used in ptlrpc_cache and large (1MB and 512KB) slab allocations. My first guess is that these large allocations relate to striped directory RPCs (i.e. OUT) and recovery, and are probably not being freed except at shutdown. There are 30k 1MB allocations consuming a whopping 30GB of memory, 400k ptlrpc_cache entries consuming 300MB of memory, and 230k 1KB allocations consuming 235MB. All of those slabs are growing continuously from startup, yet there aren't any objects (ldlm_lock, ldlm_resource, ldiskfs_inode, buffer_head, dentry, etc) that might normally grow as the node is caching a lot of data.
It might be that we are just keeping too many requests in ptlrpc_cache? In that case, we might need to add a slab shrinker for ptlrpc_cache to keep the size of this slab in check, but I'm not sure this is the root problem because it seems all of the ptlrpc_cache items are in use, so it may be that there is a stray reference on the request that isn't being put somewhere? It is also a bit sad that the cache name is ptlrpc_cache but the internal usage is request_cache. The slab cache name should be changed to ptlrpc_request_cache to match the code so that grep can find all of the relevant code at once.
Is it possible that ptlrpc_history is enabled on these nodes? That would also keep a lot of requests pinned in memory in ptlrpc_server_drop_request(). Normally this wouldn't be a problem, but if there are so many large buffers attached to the RPCs this could be causing the OOM, when smaller (e.g. 32KB) buffers wouldn't be a problem. It also isn't clear that there is a benefit to keeping the request buffers in the request history, since they are never used, so it might also be possible to release the buffers before inserting the request into the history, but I'm not sure if that is required here or not?
|
console-lola-11.log.bz2