https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3771 Lustre <p>We have ongoing problem of unreclaiming slab memory stuck in Lustre. It is different from <a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> in that unmounting Lustre FS did not release the stuck memory. Also we tried lflush and also the write technique suggested by Niu Yawei in <a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> at 15/Jan/13 8:54 AM. None worked for us.</p> <p>This is an ongoing problem and created a lot of problem in our production systems.</p> <p>I will append /proc/meminfo and a 'slabtop' output below. Let me know what other information you need.</p> <p>bridge2 /proc # cat meminfo<br/> MemTotal: 65978336 kB<br/> MemFree: 4417544 kB<br/> Buffers: 7804 kB<br/> Cached: 183036 kB<br/> SwapCached: 6068 kB<br/> Active: 101840 kB<br/> Inactive: 183404 kB<br/> Active(anon): 83648 kB<br/> Inactive(anon): 13036 kB<br/> Active(file): 18192 kB<br/> Inactive(file): 170368 kB<br/> Unevictable: 3480 kB<br/> Mlocked: 3480 kB<br/> SwapTotal: 2000052 kB<br/> SwapFree: 1669420 kB<br/> Dirty: 288 kB<br/> Writeback: 0 kB<br/> AnonPages: 92980 kB<br/> Mapped: 16964 kB<br/> Shmem: 136 kB<br/> Slab: 57633936 kB<br/> SReclaimable: 1029472 kB<br/> SUnreclaim: 56604464 kB<br/> KernelStack: 5280 kB<br/> PageTables: 15928 kB<br/> NFS_Unstable: 0 kB<br/> Bounce: 0 kB<br/> WritebackTmp: 0 kB<br/> CommitLimit: 34989220 kB<br/> Committed_AS: 737448 kB<br/> VmallocTotal: 34359738367 kB<br/> VmallocUsed: 2348084 kB<br/> VmallocChunk: 34297775112 kB<br/> HardwareCorrupted: 0 kB<br/> HugePages_Total: 0<br/> HugePages_Free: 0<br/> HugePages_Rsvd: 0<br/> HugePages_Surp: 0<br/> Hugepagesize: 2048 kB<br/> DirectMap4k: 7104 kB<br/> DirectMap2M: 67100672 kB<br/> bridge2 /proc # </p> <p>bridge2 ~ # slabtop --once</p> <p> Active / Total Objects (% used) : 2291913 / 500886088 (0.5%)<br/> Active / Total Slabs (% used) : 170870 / 14351991 (1.2%)<br/> Active / Total Caches (% used) : 151 / 249 (60.6%)<br/> Active / Total Size (% used) : 838108.56K / 53998141.57K (1.6%)<br/> Minimum / Average / Maximum Object : 0.01K / 0.11K / 4096.00K</p> <p> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME <br/> 133434868 41138 0% 0.04K 1450379 92 5801516K lovsub_page_kmem<br/> 124369720 77440 0% 0.19K 6218486 20 24873944K cl_page_kmem<br/> 115027759 41264 0% 0.05K 1493867 77 5975468K lov_page_kmem<br/> 77597568 41174 0% 0.08K 1616616 48 6466464K vvp_page_kmem<br/> 44004405 38371 0% 0.26K 2933627 15 11734508K osc_page_kmem<br/> 1558690 9106 0% 0.54K 222670 7 890680K radix_tree_node<br/> 1435785 457262 31% 0.25K 95719 15 382876K size-256<br/> 991104 24455 2% 0.50K 123888 8 495552K size-512<br/> 591420 573510 96% 0.12K 19714 30 78856K size-128<br/> 583038 507363 87% 0.06K 9882 59 39528K size-64<br/> 399080 4356 1% 0.19K 19954 20 79816K cred_jar<br/> 112112 81796 72% 0.03K 1001 112 4004K size-32<br/> 106368 106154 99% 0.08K 2216 48 8864K sysfs_dir_cache<br/> 89740 26198 29% 1.00K 22435 4 89740K size-1024<br/> 87018 1601 1% 0.62K 14503 6 58012K proc_inode_cache<br/> 53772 2845 5% 0.58K 8962 6 35848K inode_cache<br/> 44781 44746 99% 8.00K 44781 1 358248K size-8192<br/> 42700 28830 67% 0.19K 2135 20 8540K dentry<br/> 38990 2213 5% 0.79K 7798 5 31192K ext3_inode_cache<br/> 25525 24880 97% 0.78K 5105 5 20420K shmem_inode_cache<br/> 23394 16849 72% 0.18K 1114 21 4456K vm_area_struct<br/> 22340 6262 28% 0.19K 1117 20 4468K filp<br/> 20415 19243 94% 0.25K 1361 15 5444K skbuff_head_cache<br/> 19893 2152 10% 0.20K 1047 19 4188K ll_obdo_cache<br/> 15097 15006 99% 4.00K 15097 1 60388K size-4096<br/> 14076 1837 13% 0.04K 153 92 612K osc_req_kmem<br/> 12696 1448 11% 0.04K 138 92 552K lovsub_req_kmem<br/> 11684 1444 12% 0.04K 127 92 508K lov_req_kmem<br/> 10028 1477 14% 0.04K 109 92 436K ccc_req_kmem<br/> 9750 3000 30% 0.12K 325 30 1300K nfs_page</p> Server: 2.1.4, centos 6.3<br/> Client: 2.1.5, sles11sp1 LU-3771

stuck 56G of SUnreclaim memory

Bug Major Resolved Not a Bug Niu Yawei Jay Lan Fri, 16 Aug 2013 19:01:40 +0000 Thu, 3 Oct 2013 16:36:55 +0000 Wed, 4 Sep 2013 02:20:25 +0000 0 8 <p>Niu</p> <p>Here is some further information from NASA. What do you advise?</p> <p>Thanks</p> <p>Peter</p> <p>The memory is consumed by the slabs, and those slabs will be destroyed when unloading lustre modules (obdclass, lov, ...), could you try to see if unloading lustre modules will help? Thanks.</p> <p>This is all CLIO metadata structures in the slabs. It also seems unusual that the number of slab objects is over 150M, which would be enough for over 600GB of pages and totally unreasonable for a node with 64GB of RAM.</p> <p>Are there any patches that have been landed after 2.1 that might fix this problem? </p> <p>Those slabs have been removed by 3bffa4d32bc5b0bc71ba6873e262ddbca436bae1 (<a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>) in master.</p> <p>I tend to think this shows a `problem' of slab allocation. Let's take a look at the first line:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>133434868 41138 0% 0.04K 1450379 92 5801516K lovsub_page_kmem </pre> </div></div> <p>so it allocated a huge number of pages which can contain 133434868 of lovsub_page{} coming to 5G memory. However only 41138 of lovsub_page{} are being actively used.</p> <p>Hi Jay, have you ever seen this caused any real problem? Usually kernel tends to cache slab object until the memory is in pressure. Therefore it is okay if the memory being used by slab can be freed later on.</p> <p>Niu,</p> <p>Yes, unloading the modules freed up the SUnreclaim slabs. </p> <p>The systems have 62G memory. One system constantly has high number of memory in slab (&gt; 45G), most of them in SUnreclaim (~ 99%). The past two days I checked many times, and found the "Active / Total Slabes (% used)" showed between 4.5% ~ 8.5% of slab usage. I was told when the usage dropped below, say 1.5%, the system would be very sluggish and unusable.</p> <p>45G x (100 - 8)% = 41.4G of SUnreclaim memory unused is a lot. It is OK if the memory just park there while we still have enough memory for normal operation, but they should be freed up when the system need them.</p> <p>I would cherry-pick 3bffa4d (<a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>) into our 2.1.5. Do you know it would solve our problem? Thanks!</p> <p>It seems to me patch "<a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> clio: save memory allocations for cl_page" decreases memory allocation of clio from 6 down to 2. It certainly will ease the pressure, but it does not seem to address the problem of memory stuck in SUnreclaim.</p> <p>patch 3bffa4d would mitigate the problem a little bit because less slab data structures will be used for a page, but that is definitely not a fix. Actually we can't do anything for that because it's up to linux kernel vm management for when to free those memory.</p> <p>Niu, probably we should take a look at slab implementation to check if there is any tunable parameters for this.</p> <p>There is /proc/sys/vm/min_slab_ratio in linux kernel and default is 5, you may set it higher and see if it can help.</p> <p>The min_slab_ratio defines the threshold when the kernel will free the reclaimable slab. But in our cases, the slabs held up by Lustre were in Unreclaimable slabs. Changing that value would not help.</p> <p><a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> found a case where unreclaimable slabs should have been released but not. We may hit another case? Don't know. But, as Andreas commented those number were unreasonably high.</p> <p>Yes, you're right about this. The slab memory should be in SRclaimable but it was in SUnreclaim for unknown reason.</p> <p>Based on the low number of `active objs' in slabinfo, it doesn't look like a memory leak problem - was the memory all released after unloading lustre modules?</p> <blockquote> <p>Yes, you're right about this. The slab memory should be in SRclaimable but it was in SUnreclaim for unknown reason.</p></blockquote> <p>The slab memory is accounted in SUnreclaim when the slab cache is created without SLAB_RECLAIM_ACCOUNT flag, the cl/lov/osc page slabs are created without this flag, so they showed in SUnreclaim, and I think adding the flag and shrinker callback won't help, because the problem now is that slab cache isn't reaped but not the slab objects are not freed. </p> <blockquote> <p>Based on the low number of `active objs' in slabinfo, it doesn't look like a memory leak problem - was the memory all released after unloading lustre modules?</p></blockquote> <p>Right, it's not a memory leak problem, and all the slab memory will be freed after unloading lustre modules (see Jay's previous comment)</p> <p>I don't think it's a lustre problem, the slab objects are already freed and put back in the slab cache after umount, so the problem is that kernel didn't reap the slab cache for some reason (actually, I don't know how to reap slab cache initiatively in 2.6 kernel).</p> <blockquote> <p><a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> found a case where unreclaimable slabs should have been released but not. We may hit another case? Don't know. But, as Andreas commented those number were unreasonably high.</p></blockquote> <p><a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a> is totally different, in <a href="https://jira.whamcloud.com/browse/LU-2613" title="opening and closing file can generate &#39;unreclaimable slab&#39; space" class="issue-link" data-issue-key="LU-2613"><del>LU-2613</del></a>, Lustre holds the slab objects, in this ticket, Lustre doesn't hold any object (fs is already umount, and all objects freed), but kernel doesn't free memory in slab cache.</p> <p>I think the reason of such a high object number is that the filesystem has been umount/umount &amp; used for a very long time, so lots of objects were created.</p> <p>Niu, it is not exactly as what you said "the slab objects are already freed and put back in the slab cache after umount".</p> <p>Bridge2 was last rebooted 2 days ago at Aug 19 04:38. All 8 lustre fs have not been umounted since. Here is the 'slabtop' output:</p> <p> Active / Total Objects (% used) : 8844277 / 385193960 (2.3%)<br/> Active / Total Slabs (% used) : 411903 / 9494044 (4.3%)<br/> Active / Total Caches (% used) : 151 / 249 (60.6%)<br/> Active / Total Size (% used) : 1729034.95K / 35916957.66K (4.8%)<br/> Minimum / Average / Maximum Object : 0.01K / 0.09K / 4096.00K</p> <p> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME <br/> 124673156 1052956 0% 0.04K 1355143 92 5420572K lovsub_page_kmem</p> <p>104933521 1057165 1% 0.05K 1362773 77 5451092K lov_page_kmem</p> <p>68093040 1050624 1% 0.08K 1418605 48 5674420K vvp_page_kmem</p> <p>53508260 2119860 3% 0.19K 2675413 20 10701652K cl_page_kmem</p> <p>12914768 27449 0% 0.50K 1614346 8 6457384K size-512</p> <p>11721315 1058247 9% 0.26K 781421 15 3125684K osc_page_kmem</p> <p>5503680 81827 1% 0.03K 49140 112 196560K size-32<br/> 1856400 1690306 91% 0.12K 61880 30 247520K size-128</p> <p>639760 4178 0% 0.19K 31988 20 127952K cred_jar<br/> 475658 124818 26% 0.06K 8062 59 32248K size-64<br/> 250560 87321 34% 0.25K 16704 15 66816K size-256<br/> 106416 106154 99% 0.08K 2217 48 8868K sysfs_dir_cache<br/> 65527 65103 99% 0.54K 9361 7 37444K radix_tree_node<br/> 46062 46027 99% 8.00K 46062 1 368496K size-8192<br/> 43160 41625 96% 0.19K 2158 20 8632K dentry<br/> 42600 35980 84% 1.00K 10650 4 42600K size-1024<br/> 37962 11173 29% 0.10K 1026 37 4104K buffer_head<br/> 28440 1072 3% 0.12K 948 30 3792K nfs_page<br/> 27798 5355 19% 0.58K 4633 6 18532K inode_cache<br/> 25545 24886 97% 0.78K 5109 5 20436K shmem_inode_cache<br/> 20910 19187 91% 0.25K 1394 15 5576K skbuff_head_cache<br/> 17346 4255 24% 0.62K 2891 6 11564K proc_inode_cache<br/> 15049 14950 99% 4.00K 15049 1 60196K size-4096<br/> 13902 12067 86% 0.18K 662 21 2648K vm_area_struct<br/> 10200 6255 61% 0.19K 510 20 2040K filp<br/> 8408 8192 97% 0.44K 1051 8 4204K ib_mad<br/> 5900 5586 94% 2.00K 2950 2 11800K size-2048<br/> 4445 3003 67% 0.56K 635 7 2540K ldlm_locks<br/> 4032 2810 69% 0.02K 28 144 112K anon_vma<br/> 3696 3360 90% 0.08K 77 48 308K Acpi-State<br/> 3638 2106 57% 0.11K 107 34 428K journal_head<br/> 3498 2873 82% 0.07K 66 53 264K Acpi-Operand<br/> 3312 72 2% 0.02K 23 144 92K journal_handle<br/> 3017 2366 78% 0.50K 431 7 1724K skbuff_fclone_cache</p> <p>We have not done umount of Lustre fs. From past observation unmount would not free up the slab memory until we unload lustre modules. The fact that unloading the lustre modules frees up slabs suggests some communication not right between kernel and the lustre modules. Why and how? I do not know.</p> <blockquote> <p>We have not done umount of Lustre fs. From past observation unmount would not free up the slab memory until we unload lustre modules. The fact that unloading the lustre modules frees up slabs suggests some communication not right between kernel and the lustre modules. Why and how? I do not know.</p></blockquote> <p>Lustre should have freed all slab objects (done by kmem_cache_free()) after umount, but it doesn't mean slab cache will free the memory used by object immediately, slab cache will still hold the memory for next use, the memory will only be freed when slab thinks memory is tight or the slab is destroyed. (when unload Lustre module, the slabs will be destroyed)</p> <p>If the slab cache consumed too much memory, and that result in unusable/sluggish system, I think there could be some defects in the slab reap mechanism (slab cache is run by kernel, not Lustre), what we can do is to reduce use of slab in Lustre, the fix of <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> could be helpful.</p> <p>I now think the problem was probably caused by a certain application by certain user(s). For about a week after crash, about 90% of system memory were in slab. Last Friday I checked again, the slab percentage dropped down to 38%. Today the slab percentage was 30%.</p> <p>We can close this ticket. Next time should the problem happen again, we will track down the user and help him/her figure out how to address the problem.</p> <p>ok - thanks Jay!</p> Duplicate LU-4053 Development Rank 1|hzvy5b: Rank (Obsolete) 9716 Severity