https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7780 Lustre <p>Error happened during soak testing of build '20160215' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20150215" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20150215</a>). DNE is enabled.<br/> MDT had been formatted using ldiskfs, OSTs using zfs. MDS nodes are configured in active-active HA failover configuration.</p> <p>Please note that build 20150215 is a vanilla build of the master brunch.<br/> This issue might be addressed by the changes included in build '20160210' as we didn't observe this issue in a two day test session.</p> <ul> <li>2016-02-15 15:37:51,169:fsmgmt.fsmgmt:INFO triggering fault mds_failover (for <tt>lola-11</tt>)</li> <li>2016-02-15 15:44:57,839:fsmgmt.fsmgmt:INFO mds_failover just completed (for <tt>lola-11</tt>)</li> <li>After that the slabs memory consumption of slabs continuously increased till all resources are exhausted at 2016-02-15 22:38.</li> <li>Most pages are allocated by <em>size-1048576</em> slabs. High score list reads as <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct 20160215 22:46:20 size-1048576 29147 30562844672 29147 30562844672 29147 30562844672 29147 30562844672 0 0 20160215 22:46:20 size-262144 1793 470024192 1793 470024192 1793 470024192 1793 470024192 0 0 20160215 22:46:20 ptlrpc_cache 399364 306711552 399380 306723840 79873 327159808 79876 327172096 180224 0 20160215 22:46:20 size-1024 229179 234679296 229188 234688512 57295 234680320 57297 234688512 -24576 0 20160215 22:46:20 size-512 256540 131348480 258232 132214784 32278 132210688 32279 132214784 86016 0 20160215 22:46:20 size-192 460848 88482816 460880 88488960 23043 94384128 23044 94388224 28672 0 20160215 22:46:20 size-8192 5776 47316992 5776 47316992 5776 47316992 5776 47316992 -8192 0 20160215 22:46:20 size-128 265120 33935360 266250 34080000 8875 36352000 8875 36352000 0 0 20160215 22:46:20 size-65536 361 23658496 361 23658496 361 23658496 361 23658496 0 0 20160215 22:46:20 kmem_cache 289 9506944 289 9506944 289 18939904 289 18939904 0 0 </pre> </div></div> <p>(see attached file slab-usage-by-allocation-descending.dat)</p></li> </ul> <p>Attached files messages, console file of <tt>lola-11</tt>. Sorted slab usage as oom-killer was started.</p> lola<br/> &nbsp;build: 2.8.50-6-gf9ca359 ; commit f9ca359284357d145819beb08b316e932f7a3060 LU-7780

MDS crashed with oom-killer

Bug Critical Resolved Duplicate Frank Heckes Frank Heckes soak Tue, 16 Feb 2016 11:46:43 +0000 Mon, 6 Jun 2016 15:52:25 +0000 Wed, 16 Mar 2016 19:27:29 +0000 0 7 <p>Three (different) MDSes crashed with oom-killer started over the weekend (2016-02-20 - 2016-02-21).<br/> Performance counters and event sequence can be provided if necessary</p> <p>Can you please do the tip of b2_8 testign to see if it's also affected?</p> <p>Also on master - if you can enable memory allocation tracing, reproduce, and then extract the debug log out of a crashdump so that we can see where are all of those allocations come from.</p> <p>currently '+malloc' has been added to thed debug on server nodes</p> <p>Frank or Cliff,<br/> it might be useful to increase the debug buffer size and turn on trace logging (<tt>lctl set_param debug_mb=128 debug=+trace</tt>) and then let it continue to run for a minute or two and dump the debug log without having to wait for the crash to be hit. The <tt>size-1024</tt> items are being allocated 10/sec so the cause will appear in any logs. Similarly, the <tt>size-1048576</tt> allocations are done every minute so the cause should also be in any logs dumped. The <tt>size-262144</tt> is only being allocated in chunks every 15 minutes, so it might be related to recovery if that is the frequency of failovers, so that might need to be timed to coordinate with a failover or similar, or the size of the log increased to 512 or 1024 MB.</p> <p>Di, the memory is all used in ptlrpc_cache and large (1MB and 512KB) slab allocations. My first guess is that these large allocations relate to striped directory RPCs (i.e. OUT) and recovery, and are probably not being freed except at shutdown. There are 30k 1MB allocations consuming a whopping 30GB of memory, 400k ptlrpc_cache entries consuming 300MB of memory, and 230k 1KB allocations consuming 235MB. All of those slabs are growing continuously from startup, yet there aren't any objects (ldlm_lock, ldlm_resource, ldiskfs_inode, buffer_head, dentry, etc) that might normally grow as the node is caching a lot of data.</p> <p>It might be that we are just keeping too many requests in <tt>ptlrpc_cache</tt>? In that case, we might need to add a slab shrinker for ptlrpc_cache to keep the size of this slab in check, but I'm not sure this is the root problem because it seems all of the <tt>ptlrpc_cache</tt> items are in use, so it may be that there is a stray reference on the request that isn't being put somewhere? It is also a bit sad that the cache name is <tt>ptlrpc_cache</tt> but the internal usage is <tt>request_cache</tt>. The slab cache name should be changed to <tt>ptlrpc_request_cache</tt> to match the code so that grep can find all of the relevant code at once.</p> <p>Is it possible that ptlrpc_history is enabled on these nodes? That would also keep a lot of requests pinned in memory in <tt>ptlrpc_server_drop_request()</tt>. Normally this wouldn't be a problem, but if there are so many large buffers attached to the RPCs this could be causing the OOM, when smaller (e.g. 32KB) buffers wouldn't be a problem. It also isn't clear that there is a benefit to keeping the request buffers in the request history, since they are never used, so it might also be possible to release the buffers before inserting the request into the history, but I'm not sure if that is required here or not?</p> <p>Debug filter has been changed to : <tt>debug_mb=128 debug=+malloc +trace</tt></p> <p>Moving to 2.9 because it seems that this issue only occurs with multiple MDTs per MDS and does not happen with the more common configuration of a single MDT per MDS. Is this a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7836" title="MDSes crashed with oom-killer" class="issue-link" data-issue-key="LU-7836"><del>LU-7836</del></a>?</p> <p>Yes, this is duplicate with lu-7836</p> <p>ok thanks Di</p> Related LU-7836 Development Rank 1|hzy1cf: Rank (Obsolete) 9223372036854775807 Severity