https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7517 Lustre <p>The error below happens during soak testing of change 16838 patch set #31 (no Wiki entry for build exits, yet) on cluster lola. DNE is enabled and MDSes are configured in active-active HA failover configuration.</p> <p> Primary resources of MDT lola-11 were failed back at Dec, 3 20:18.<br/> The allocation of slabs increased continuously till ~ 31 GB till crash<br/> MDS node lola-11 crashed with oom-killer at Dec, 4 00:21 (local time). (see also <a href="https://jira.whamcloud.com/browse/LU-7432" title="oom-killer started on MDSes" class="issue-link" data-issue-key="LU-7432"><del>LU-7432</del></a>)<br/> ptlrpc_cache seems to be the biggest consumer<br/> Attached lola-11's messages, console log, vmcore-dmesg file, collectl (version V4.0.2-1) files (for time interval specified above). Also<br/> attached files containing extracted counters for memory, slab totals and per slab allocation.</p> lola:<br/> build: tip of master + #31 of change 16383 LU-7517

oom killer active after failback of MDS resources

Bug Blocker Resolved Won't Fix WC Triage Frank Heckes soak Fri, 4 Dec 2015 15:14:30 +0000 Tue, 24 Jan 2017 22:40:05 +0000 Tue, 24 Jan 2017 22:40:05 +0000 0 3 <p>The crash dump has been saved to lola-1:/scratch/crashdumps/lu-7517/127.0.0.1-2015-12-04-00\:22\:36.<br/> It turned out that the collectl raw files are to big to be uploaded to Jira. I saved them to lola-1:/scratch/crashdumps/lu-7517.</p> <p>No debug log files have been written.</p> <p>looks like lola-8 and lola-9 got OOM as well.</p> <p>It looks like most the memory is holden by 1M size slab</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>20151204 00:21:00 size-1048576 29758 31203524608 29758 31203524608 29758 31203524608 29758 31203524608 0 0 </pre> </div></div> <p>The error showed up on all soak MDSes (lola-8 not reported in detail) running <tt>soak</tt> for build <a href="https://build.hpdd.intel.com/job/lustre-reviews/36192/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/36192/</a></p> <ul> <li>lola-9 <ul> <li>20151213 09:52:40 failback MDTS to lola finished</li> <li>Dec 13 16:05:01 lola-9 oom-killer started</li> </ul> </li> <li>lola-10 <ul> <li>20151213 08:01:40 failback MDTs to lola-10 finished</li> <li>Dec 13 11:40:02 lola-10 oom-killer started</li> </ul> </li> <li>lola-11 <ul> <li>20151214 02:15:00 failback of MDTs to lola-11 finished</li> <li>Dec 14 12:05:01 lola-11 oom-killer started</li> </ul> </li> </ul> <p>From failback till start of oom-killer the <tt>size-1048576</tt> slabs continuously increased and are the biggest memory consumers:<br/> <b>lola-9</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct slab-details/size-1048576.dat:20151213 16:05:40 size-1048576 23122 24245174272 23122 24245174272 23122 24245174272 23122 24245174272 0 0 slab-details/size-512.dat:20151213 16:05:40 size-512 11151621 5709629952 11152824 5710245888 1394094 5710209024 1394103 5710245888 7835648 0 slab-details/size-128.dat:20151213 16:05:40 size-128 7076064 905736192 7077360 905902080 235911 966291456 235912 966295552 1376256 0 slab-details/size-262144.dat:20151213 16:05:40 size-262144 1673 438566912 1673 438566912 1673 438566912 1673 438566912 0 0 slab-details/ptlrpc_cache.dat:20151213 16:05:40 ptlrpc_cache 376940 289489920 376940 289489920 75388 308789248 75388 308789248 204800 0 ... </pre> </div></div> <p><b>lola-10</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct slab-details/size-1048576.dat:20151213 11:40:40 size-1048576 29494 30926700544 29494 30926700544 29494 30926700544 29494 30926700544 0 0 slab-details/size-262144.dat:20151213 11:40:40 size-262144 1015 266076160 1015 266076160 1015 266076160 1015 266076160 0 0 slab-details/ptlrpc_cache.dat:20151213 11:40:40 ptlrpc_cache 195920 150466560 195920 150466560 39184 160497664 39184 160497664 167936 0 slab-details/size-1024.dat:20151213 11:40:40 size-1024 133537 136741888 133540 136744960 33385 136744960 33385 136744960 73728 0 slab-details/size-512.dat:20151213 11:40:40 size-512 150577 77095424 153896 78794752 19237 78794752 19237 78794752 0 0 ... </pre> </div></div> <p><b>lola-11</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct slab-details/size-1048576.dat:20151214 12:05:40 size-1048576 29392 30819745792 29392 30819745792 29392 30819745792 29392 30819745792 0 0 slab-details/size-262144.dat:20151214 12:05:40 size-262144 1345 352583680 1345 352583680 1345 352583680 1345 352583680 0 0 slab-details/ptlrpc_cache.dat:20151214 12:05:40 ptlrpc_cache 224232 172210176 224290 172254720 44858 183738368 44858 183738368 57344 0 slab-details/size-1024.dat:20151214 12:05:40 size-1024 150612 154226688 150632 154247168 37655 154234880 37658 154247168 -20480 0 slab-details/size-8192.dat:20151214 12:05:40 size-8192 8859 72572928 8859 72572928 8859 72572928 8859 72572928 0 0 ... </pre> </div></div> <p>Attached messages, console log files and extracted <tt>collectl</tt> counters for memory, slab-global, slab-details for each node.</p> <p>Error appeared again on a MDT for build '20151214' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151214" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151214</a>)</p> <p>Old issue not reproduced on recent builds</p> Development Rank 1|hzxuxj: Rank (Obsolete) 9223372036854775807 Severity