https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4264 Lustre <p>NOAA has been having a problem with OOM on their OSSes causing failover. Looking at collectl output from right before the crash, it appears that all the memory is being consumed by the size-256 slab:<br/> size-256 168M 41122M 168M 41122M 11229K 43864M 11229K 43864M 200704 0.0</p> <p>Is there a way to determine what those objects are and reduce the amount of memory they are taking? The vmcore is available if necessary. </p> LU-4264

Excessive slab usage on 1.8.9 server

Bug Critical Resolved Fixed Oleg Drokin Oz Rentas Mon, 18 Nov 2013 14:28:05 +0000 Fri, 1 Sep 2017 14:55:31 +0000 Fri, 1 Sep 2017 14:55:31 +0000 Lustre 1.8.9 1 8 <p>You can do "echo +malloc &gt;/proc/sys/lnet/debug" (no quotes) on the affected servers, and then as the number keeps growing, you can do lctl dk &gt;/tmp/somewhere<br/> This will tell you what allocations (and freeings) lustre has been doing. But if the allocation does not originate from Lustre, this won't tell you anything.</p> <p>You can also enable in-kernel memory leak tracer if you suspect you have a genuine memory leak (it's a kernel config option in debug options).<br/> this will print warnings in kernel dmesg log, and you will be able to get suspected leaked allocations and backtraces that led to it from a special debugfs file.</p> <p>The log from crash and the collectl from the time of the OOM. </p> <p>I found an oss whose slab appears to be growing larger than the other OSSes. oss-2-18 is typical </p> <ol> <li>pdsh -a cat /proc/slabinfo |grep size-256<br/> lfs-oss-2-18: size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0<br/> lfs-oss-2-18: size-256 1633652 1646400 256 15 1 : tunables 120 60 8 : slabdata 109760 109760 76<br/> ...<br/> lfs-oss-2-15: size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0<br/> lfs-oss-2-15: size-256 168811996 168812010 256 15 1 : tunables 120 60 8 : slabdata 11254134 11254134 39</li> </ol> <p>So there is a couple orders of magnitude difference, and it keeps increasing.</p> <p>Slab size-256 allocations within a minute or two:</p> <p><span class="error">&#91;root@lfs-oss-2-15 ~&#93;</span># cat /proc/slabinfo |grep e-256<br/> size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0<br/> size-256 168812385 168812475 256 15 1 : tunables 120 60 8 : slabdata 11254165 11254165 376</p> <p><span class="error">&#91;root@lfs-oss-2-15 ~&#93;</span># cat /proc/slabinfo |grep e-256<br/> size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0<br/> size-256 168813438 168813450 256 15 1 : tunables 120 60 8 : slabdata 11254230 11254230 27</p> <p>I ran malloc and grepped for slab-alloc:<br/> 00010000:00000010:15:1384787443.947162:0:24815:0:(ldlm_lock.c:338:ldlm_lock_new()) slab-alloced 'lock': 512 at ffff810abf0b8400.<br/> 00010000:00000010:15:1384787443.947167:0:24815:0:(ldlm_extent.c:799:ldlm_interval_alloc()) slab-alloced 'node': 72 at ffff810b4013c1c0.<br/> 00010000:00000010:15:1384787444.003597:0:24908:0:(ldlm_resource.c:757:ldlm_resource_new()) slab-alloced 'res': 344 at ffff8109e26d0cc0.<br/> 00002000:00000010:15:1384787444.197912:0:5041:0:(fsfilt-ldiskfs.c:809:fsfilt_ldiskfs_add_journal_cb()) slab-alloced 'fcb': 56 at ffff810ae623be98.</p> <p>I checked the number of locks to see if they were very large, but it is comparable to other OSSes:<br/> <span class="error">&#91;root@lfs-oss-2-15 ~&#93;</span># cat /proc/fs/lustre/ldlm/namespaces/filter-scratch2-OST*/lock_count<br/> 22085<br/> 9504<br/> 22975<br/> 12472<br/> 7644<br/> 14597<br/> 6662<br/> 8624<br/> 20551<br/> 8725<br/> 16634</p> <p>Any ideas? The OSSes keep crashing.</p> <p>Attached the malloc dk log</p> <p>It appears to be a memory leak of some kind - the servers having issues had uptimes of 155 days, while the other OSSes were more recently rebooted. I wasn't able to determine where the leak was. I looked at some of the objects in the slab of a vmcore we got, and some were definitely Lustre related, but I couldn't narrow it down to anything. I also tried unloading modules and stopping processes to try to get the memory back, but it didn't budge. </p> <p>Any tips for debugging this kind of problem?</p> <p>I didn't find leak information from the log file of dk1 you posted. How much time would it survive for each restart? The log file has collected the memory allocation from lustre for 80 seconds, so it must have hit thousands of times if the leaking allocation is from lustre.</p> <p>256 byte-size slab is created by linux kernel to serve generic kmalloc, and lustre creates its own slab cache for dlm lock, therefore it won't be the lak from dlm lock fore sure. I used the following command to check the log to try to find the leaking blocks:</p> <p>egrep 'kmalloced|kfreed' dk1 |awk '</p> {if($4&gt;128 &amp;&amp; $4&lt;=256)print} <p>' &gt; temp_log<br/> perl ~/srcs/lustre/lustre/tests/leak_finder.pl temp_log</p> <p>(you can find leak_finder from lustre-tests rpm).</p> <p>the output was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ... freed 144 bytes at ffff810a83297980 called desc (client.c:ptlrpc_free_bulk:188) freed 144 bytes at ffff8107f9ca16c0 called desc (client.c:ptlrpc_free_bulk:188) freed 144 bytes at ffff810825d80980 called desc (client.c:ptlrpc_free_bulk:188) freed 144 bytes at ffff81090a21d1c0 called desc (client.c:ptlrpc_free_bulk:188) malloced 176 bytes at ffff810accc66880 called blwi (ldlm_lockd.c:ldlm_bl_to_thread:1672) *** Leak: 176 bytes allocated at ffff810accc66880 (ldlm_lockd.c:ldlm_bl_to_thread:1672, debug file line 15320) maximum used: 4912, amount leaked: 176 </pre> </div></div> <p>the last entry is not a true memory leak, because the free log was not yet collected.</p> <p>Is this node different from the others in anyways, kernel, drivers or recent updates? This is probably not a Lustre problem.</p> <p>Anyway, let's drill down a little bit. Let's try ftrace to see if we can find it out.</p> <p>Please follow the instruction here: <a href="http://elinux.org/Kernel_dynamic_memory_analysis" class="external-link" target="_blank" rel="nofollow noopener">http://elinux.org/Kernel_dynamic_memory_analysis</a>, please read the Debugfs section and then go to Dynamic section.</p> <p>You can trace kmalloc and kfree events as follows:</p> <p>echo "kmem:kmalloc_node kmem:kfree kmem:kmalloc" &gt; /sys/kernel/debug/tracing/set_event</p> <p>Then enable the trace by:</p> <p>echo "1" &gt; /sys/kernel/debug/tracing/tracing_on</p> <p>After a while, you believe you have seen memory reduction, then you can dump the memory allocation information by:</p> <p>echo "0" &gt; /sys/kernel/debug/tracing/tracing_on<br/> cat /sys/kernel/debug/tracing/trace &gt; trace_log</p> <p>Also, you need to dump the kernel symbol of running kernel by:</p> <p>cat /proc/kallsyms &gt; kallsyms.txt</p> <p>Then we can do further analysis.</p> <p>Unfortunately, it looks like RHEL5 doesn't have trace support compiled in, so we would have to reboot.</p> <p>The memory leak seems to be occurring on all the servers, we still have one system up that has been up for 155 days and it has very high memory usage in slab-256. It is the backup MDS, so we have not rebooted it yet, in case there is information we can still get. </p> <p>It seems that the servers go about 150 days before starting to have problems. I thought that these servers had already been reboot earlier today, but that was not the case. The file system is mostly stable now, so the severity can be reduced. NOAA is very anxious about the memory leaks, however, and so we still need to figure out where that is coming from.</p> <p>Well, given that you cannot identify a culprit in the lustre log (or is it a really-really slow leak?) and it takes 150 days to manifest, also assuming you don't want to recompile your kernel in a way that would allow tracking leaks, the only realistic options left to you, I guess are:</p> <p>schedule some quiet time and reboot the remaining OSSes with 150+ days uptime, and reboot them every 100 days or so until you can gather some extra debug or upgrade to a newer version on those nodes (so that reboots are controllable vs random).</p> <p>Just do nothing and the servers will fail when they fail and recover all by themselves.</p> <p>Ok, that makes sense.</p> <p>Is the vmcore useful at all in identifying what is occupying the memory?</p> <p>no, vmcore won't help this case.</p> <p>I think I found the memory leak, it looks like a Mellanox patch was incorrectly backported. I created a Redhat ticket here:<br/> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1049601" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1049601</a></p> <p>I'll attach the original 2.6.18-308.11.1 version and the broken 2.6.18-348.1.1. Can you tell me if my analysis makes sense?</p> <p>Here's a link to the kernel.org git version:<br/> <a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/hw/mlx4/main.c" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/hw/mlx4/main.c</a></p> <p>ib_link_query_port is the function with the memory leak.</p> <p>I have a similar problem here. Same kernel and slow slab usage increase over time. I do not have access to the Bugzilla link. By looking at the main.c differences, I can't pinpoint the leak. Can you provide a patch or a link to the commit that introduced problem?</p> <p>oh weird, I wonder why it's private. I'll attach the patch, and try to get it into Gerrit.</p> <p><a href="http://review.whamcloud.com/#/c/8942/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8942/</a></p> <p>This is resolved. Please close.</p> <p>ok thanks</p> Development End date Fri, 9 May 2014 14:28:05 +0000 Rank 1|hzw9kv: Rank (Obsolete) 11714 Severity Start date Mon, 18 Nov 2013 14:28:05 +0000