https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3897 Lustre <p>Hi,</p> <p>Several Bull customers running lustre 2.1.x had a hang in ldlm_pools_shrink on a lustre client (login node).<br/> The system was hung and a dump was initiated from the the bmc, by sending a NMI.<br/> The dump shows there was no more activity on the system. The 12 CPUs are idle (swapper).<br/> A lot of processes are in page_fault(), blocked in ldlm_pools_shrink().<br/> I have attached the output of the "foreach bt" crash command. Let me know if you need the vmcore file.</p> <p>Each time, we can see a lot of OOM messages in the syslog of the dump files.</p> <p>This issue looks like <a href="https://jira.whamcloud.com/browse/LU-2468" title="MDS out of memory, blocked in ldlm_pools_shrink()" class="issue-link" data-issue-key="LU-2468"><del>LU-2468</del></a>.</p> <p>Thanks,<br/> Sebastien.</p> LU-3897

Hang up in ldlm_pools_shrink under OOM

Bug Major Resolved Not a Bug Zhenyu Xu Sebastien Buisson Fri, 6 Sep 2013 14:02:33 +0000 Thu, 5 Dec 2013 18:11:05 +0000 Thu, 5 Dec 2013 18:11:04 +0000 Lustre 2.1.3 0 6 <p>Bobijam</p> <p>What do you recommend here?</p> <p>Thanks</p> <p>Peter</p> <p>Oleg, </p> <p>Would you consider <a href="http://review.whamcloud.com/#/c/4954/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/4954/</a> ？I don't know whether Vitaly has addressed the issue somewhere.</p> <p>I dout that patch would help.</p> <p>Sebastien, can we get kernel dmesg log please, I wonder what's there/</p> <p>It seems that something forgot to release namespace lock on the client (at least I don't see anything that's in an area guarded by this lock, everybody seems to wait to acquire it), possibly there's some place that forgets to release it on an error exit path and here's hope there's some clue in the dmesg.</p> <p>Hi,</p> <p>This is the dmesg requested by Oleg, taken from the crash dump.</p> <p>Sebastien.</p> <p>hmm, why the dmesg does not have any lustre log informations? I just saw call trace and mem-info report in it.</p> <p>Probably there is so many information dumped in the dmesg regarding the OOM issue that it replaces the Lustre logs.</p> <p>From the dump, do you know how we can access the memory containing the Lustre debug logs?</p> <p>I haven't find relevant info to get the root cause. </p> <p>What's the memory usage on the clients? I wonder whether it is the client uses too much memory for lock or for data cache.</p> <p>what's the "lctl get_param ldlm.namespaces.*.lru_size" values?</p> <p>Hi,</p> <p>Not too easy to get the lru_size from the crash dump.<br/> We dumped the 'struct obd_device' in the obd_devs array, then the 'struct obd_namespace" to reach the 'ns_nr_unused' field corresponding to the lru_size information.</p> <p>All values are 0.</p> <p>HTH,<br/> Sebastien.</p> <p>If lru_size is 0, it means lock does not take much memory, what is the "lctl get_param llite.*.max_cached_mb" output? Can you set it to a smaller value to keep less data cache on the client side?</p> <p>Hi,</p> <p>In the crash dump I looked at the value of ll_async_page_max, which is 12365796, so max_cached_mb is around 48 GB, ie 3/4 of total node memory.</p> <p>But in any case, remember there is a regression in CLIO in 2.1 causing this max_cached_mb value to be never used!</p> <p>So there is no means to limit Lustre data cache <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Sebastien.</p> <p>Hi Sebastien, can you please show me the output of `slabtop'?</p> <p>Hi,</p> <p>I have no idea how I can run slabtop inside crash :/<br/> I have uploaded a tarball to the ftp server, under uploads/<a href="https://jira.whamcloud.com/browse/LU-3897" title="Hang up in ldlm_pools_shrink under OOM" class="issue-link" data-issue-key="LU-3897"><del>LU-3897</del></a>. It contains the vmcore, the vmlinux, and the Lustre modules.</p> <p>Maybe you will be able to get the information you are looking for.</p> <p>Thanks,<br/> Sebastien.</p> <p>`kmem -s' will print out slabinfo in this case. Thanks for the crashdump, and I will take a look.</p> <p>the system ran out of memory, from the output of `kmem -i':</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 16487729 62.9 GB ---- FREE 37266 145.6 MB 0% of TOTAL MEM USED 16450463 62.8 GB 99% of TOTAL MEM SHARED 401 1.6 MB 0% of TOTAL MEM BUFFERS 99 396 KB 0% of TOTAL MEM CACHED 57 228 KB 0% of TOTAL MEM SLAB 22513 87.9 MB 0% of TOTAL MEM TOTAL SWAP 255857 999.4 MB ---- SWAP USED 255856 999.4 MB 99% of TOTAL SWAP SWAP FREE 1 4 KB 0% of TOTAL SWAP </pre> </div></div> <p>Both system memory and swap space were used up. Page cache and slab cache were normal, no excessive usage at all.</p> <p>It turned out most of the memory were used for anonymous memory mapping.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; kmem -V VM_STAT: NR_FREE_PAGES: 37266 NR_INACTIVE_ANON: 956162 NR_ACTIVE_ANON: 15158596 NR_INACTIVE_FILE: 0 NR_ACTIVE_FILE: 99 NR_UNEVICTABLE: 0 NR_MLOCK: 0 NR_ANON_PAGES: 16115037 </pre> </div></div> <p>there was 16115037 NR_ANON_PAGES which is 61G in size. I guess the application is probably using too much memory or badly written. I don't think we can help in this case.</p> <p>Please reopen this ticket if you have more questions</p> Development Rank 1|hzw0nr: Rank (Obsolete) 10187 Severity