https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7836 Lustre <p>Error occurred during soak testing of build '20160302' (b2_8 RC4) (see: <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160302" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160302</a> also). DNE is enabled. MDTs had been formatted using <em>ldiskfs</em>, OSTs using <em>zfs</em>. MDS nodes are configured in active - active HA failover configuration. (For teset set-up configuration see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-Configuration" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-Configuration</a>)</p> <p>Note: This might be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7780" title="MDS crashed with oom-killer" class="issue-link" data-issue-key="LU-7780"><del>LU-7780</del></a></p> <p>Sequence of events:</p> <ul> <li>2016-03-01 20:38:40 triggering fault mds_failover (lola-10 --&gt; lola-11)</li> <li>2016-03-01 20:41:58 lola-8 LNet process hang on lola-8 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: Service thread pid 5074 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes to /tmp/lustre-log.1456893718.5074 </pre> </div></div> <p>(see attached file <tt>lustre-log.1456893718.5074</tt></p></li> <li>Lustre Recovery never completed (till next failover)</li> <li>All slurm jobs are stalled and began to timeout after ~ 20:40<br/> Slabs continuously allocated memory. Top 10 consumers before next<br/> failover are <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>slab-details/size-1048576.dat:20160301 22:09:20 size-1048576 13215 13856931840 13215 13856931840 13215 13856931840 13215 13856931840 67108864 0 slab-details/size-262144.dat:20160301 22:09:20 size-262144 385 100925440 385 100925440 385 100925440 385 100925440 0 0 slab-details/size-192.dat:20160301 22:09:20 size-192 268155 51485760 268320 51517440 13416 54951936 13416 54951936 24576 0 slab-details/size-8192.dat:20160301 22:09:20 size-8192 5508 45121536 5508 45121536 5508 45121536 5508 45121536 0 0 slab-details/size-1024.dat:20160301 22:09:20 size-1024 41857 42861568 41864 42868736 10466 42868736 10466 42868736 110592 0 slab-details/ptlrpc_cache.dat:20160301 22:09:20 ptlrpc_cache 35590 27333120 35605 27344640 7121 29167616 7121 29167616 122880 0 slab-details/size-65536.dat:20160301 22:09:20 size-65536 361 23658496 361 23658496 361 23658496 361 23658496 0 0 slab-details/size-512.dat:20160301 22:09:20 size-512 37639 19271168 38920 19927040 4865 19927040 4865 19927040 45056 0 slab-details/kmem_cache.dat:20160301 22:09:20 kmem_cache 289 9506944 289 9506944 289 18939904 289 18939904 0 0 slab-details/size-4096.dat:20160301 22:09:20 size-4096 2970 12165120 2971 12169216 2970 12165120 2971 12169216 4096 0 slab-details/inode_cache.dat:20160301 22:09:20 inode_cache 15612 9242304 15678 9281376 2613 10702848 2613 10702848 0 0 </pre> </div></div></li> <li>2016-03-01 22:09:46,199:fsmgmt.fsmgmt:INFO triggering fault mds_failover (lola-10 --&gt; lola-11)</li> <li>2016-03-01 22:16:19,483:fsmgmt.fsmgmt:INFO mds_failover just completed</li> <li>Recovery of MDTs (mdt-4,5) never completed</li> <li>2016-03-02 00:15-30 Created stack traces and debug log files on <tt>lola-<span class="error">&#91;8,10,11&#93;</span></tt><br/> *2016-02-02 01:30:01 approximately at this time oom-killer started on MDS <tt>lola-10</tt><br/> Top 10 consumers are: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>slab-details/size-1048576.dat:20160302 01:32:00 size-1048576 29882 31333548032 29882 31333548032 29882 31333548032 29882 31333548032 1048576 0 slab-details/size-262144.dat:20160302 01:32:00 size-262144 641 168034304 641 168034304 641 168034304 641 168034304 0 0 slab-details/size-1024.dat:20160302 01:32:00 size-1024 84449 86475776 84484 86511616 21121 86511616 21121 86511616 106496 0 slab-details/ptlrpc_cache.dat:20160302 01:32:00 ptlrpc_cache 95492 73337856 95510 73351680 19102 78241792 19102 78241792 114688 0 slab-details/size-192.dat:20160302 01:32:00 size-192 298834 57376128 299260 57457920 14963 61288448 14963 61288448 0 0 slab-details/size-8192.dat:20160302 01:32:00 size-8192 5862 48021504 5862 48021504 5862 48021504 5862 48021504 -8192 0 slab-details/size-512.dat:20160302 01:32:00 size-512 77790 39828480 79096 40497152 9879 40464384 9887 40497152 28672 0 slab-details/size-65536.dat:20160302 01:32:00 size-65536 361 23658496 361 23658496 361 23658496 361 23658496 0 0 slab-details/kmem_cache.dat:20160302 01:32:00 kmem_cache 289 9506944 289 9506944 289 18939904 289 18939904 0 0 slab-details/size-128.dat:20160302 01:32:00 size-128 78707 10074496 100920 12917760 3364 13778944 3364 13778944 0 0 </pre> </div></div> <p>*2016-02-03 03:00 MDS <tt>lola-8</tt> also crashed with oom-killer</p></li> </ul> <p>Attached messages, console and debug logs of nodes <tt>lola-8,10,11</tt>, <br/> file <tt>recovery-status-20160302</tt> showing recovery and process status around 2016-03-02 00:15, memory and detailed slab,counters of <tt>lola-10</tt></p> lola<br/> build: <a href="https://build.hpdd.intel.com/job/lustre-b2_8/11/">https://build.hpdd.intel.com/job/lustre-b2_8/11/</a> LU-7836

MDSes crashed with oom-killer

Bug Blocker Resolved Fixed Di Wang Frank Heckes soak Wed, 2 Mar 2016 14:29:25 +0000 Fri, 5 Aug 2016 21:08:47 +0000 Fri, 5 Aug 2016 21:08:47 +0000 Lustre 2.8.0 Lustre 2.9.0 0 8 <p>slab counters of <tt>lola-8</tt> can be uploaded on demand.</p> <p>just as I asked in <a href="https://jira.whamcloud.com/browse/LU-7780" title="MDS crashed with oom-killer" class="issue-link" data-issue-key="LU-7780"><del>LU-7780</del></a> - we really need the same debug log from lustre with memory allocation tracing enabled so that we can see who is it that does the allocations, not just their sizes.</p> <p>Do you think you can collect something like that?</p> <p>you do this by adding the "malloc" debug mask to the run before the problem starts.</p> <p>Debug mask has been extended with '+malloc'</p> <p>The error didn't happened again till now.</p> <p>The issue happened again during soak testing of b2_8 RC5 (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160309" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160309</a>). <br/> Unfortunately the error occurred at the weekend (at Mar 12 01:05:02) on MDS node <tt>lola-11</tt>.<br/> Therefore th latest debug log written on the affected node <tt>lola-11</tt> only contains memory allocation messages from <b>20160311 18:08:06</b> till <b>20160311 18:34:59</b>.</p> <p>The MDS nodes, especially <tt>lola-11</tt> had been restarted (randomly) at the following times:</p> <ul> <li>mds_restart : 2016-03-11 17:13:36,111 - 2016-03-11 17:24:02,736 lola-11</li> <li>mds_restart : 2016-03-11 17:29:11,738 - 2016-03-11 17:35:40,652 lola-11</li> <li>mds_restart : 2016-03-11 19:10:02,531 - 2016-03-11 19:17:28,133 lola-11</li> <li>mds_restart : 2016-03-11 19:28:55,134 - 2016-03-11 19:39:27,661 lola-11</li> <li>mds_restart : 2016-03-11 19:45:13,663 - 2016-03-11 19:58:16,686 lola-9</li> <li>mds_restart : 2016-03-11 20:24:31,687 - 2016-03-11 20:32:39,547 lola-11</li> </ul> <p>The oom-killer on <tt>lola-11</tt> ran at 2016-03-12 01:05 after the last restart for <tt>lola-11</tt> finishing at 2016-03-11 20:32:39 (line 6 in list above).<br/> The memory debug information are from the restarts between 17:35 and 19:10 (line 2 and 3). <br/> I attached the extracted allocation messages and the original debug file and the <tt>collectl</tt> memory counters to the<br/> ticket. I hope it might contain a pointer to the code section causing the memory leak as the slab memory counter show an continuous increasing amount of allocated memory resources. Anyway, the error might haven't been triggered before the restart at 20:24 (line 6).<br/> Please let me know if any other log files are needed.</p> <p>Uploaded files are </p> <ul> <li>lola-11-lustre-loglog.20160311-183459.bz2 - full debug log</li> <li>lola-11-mem-counter-20160311_1729-1910.dat.bz2 - collectl memory counters</li> <li>lola-11-memory-debug-messages.20160311-183459.bz2 - memory allocation debug log</li> </ul> <p>The upload of file lola-11-lustre-loglog.20160311-183459.bz2 stalled every time I time half the way before completion.</p> <p>The effect of continuously increasing amount of slabs immediately occured after the clean-up (remount of MDTS on lola-10) and restart<br/> of soak during the first MDS restart of <tt>lola-10</tt>. <br/> I uploaded the debug log files dirctory <tt>lhn:/scratch/lu-7836</tt></p> <ul> <li>Till 2016-03-14 07:25 slab allocation continues to grow till ~ 9 GB</li> <li>Recovery completed at: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lola-10.log:Mar 14 07:24:35 lola-10 kernel: Lustre: soaked-MDT0005: Recovery over after 66:57, of 16 clients 16 recovered and 0 were evicted. lola-10.log:Mar 14 07:25:38 lola-10 kernel: Lustre: soaked-MDT0004: Recovery over after 67:00, of 16 clients 15 recovered and 1 was evicted. </pre> </div></div></li> <li>After that slab memory allocation remains constant at 9GB</li> <li>During 06:50 - 07:27 the following debug log were created and saved to <tt>lhn:/scratch/lu-7638</tt><br/> lustre-log.20160314-0650, lustre-log.20160314-0656, lustre-log.20160314-0714, lustre-log.20160314-0727</li> <li>umount / mount the MDTs --&gt; recovery went to state INACTIVE</li> <li>Reboot and mount of MDTs brought back MDS <tt>lola-10</tt> into state with 'wild' allocation of slabs, see:<br/> lustre-log.20160314-0801, lustre-log.20160314-0820</li> </ul> <p>Di</p> <p>Could you please look into this?</p> <p>Thanks</p> <p>Peter</p> <p>After some investigation, it looks like the MDT is blocked on update recovery, then queue too much final ping req there. I will try to make a patch.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18915" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18915</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7836" title="MDSes crashed with oom-killer" class="issue-link" data-issue-key="LU-7836"><del>LU-7836</del></a> ptlrpc: No new signal complete request<br/> Project: fs/lustre-release<br/> Branch: b2_8<br/> Current Patch Set: 1<br/> Commit: c80d98b7c1d741e239656be30b80ce723f3aaab5</p> <p>Moving to 2.9 because it seems that this issue only occurs with multiple MDTs per MDS and does not happen with the more common configuration of a single MDT per MDS. Is this a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7780" title="MDS crashed with oom-killer" class="issue-link" data-issue-key="LU-7780"><del>LU-7780</del></a>?</p> <p>Soak has been continued to execute b2_8 RC5 build with reformatted Lustre FS. <br/> Now there's only 1 MDT per MDS and 5 OSTs per OSS (unchanged). MDT had<br/> been formatted with <em>ldiskfs</em> and OSTs using <em>zfs</em>.<br/> Soak session is running for ~ 2.5 days <b>without</b> appearance of this bug.<br/> As <a href="https://jira.whamcloud.com/browse/LU-7848" title="Recovery process on MDS stalled" class="issue-link" data-issue-key="LU-7848"><del>LU-7848</del></a> is most likely related, it's worth mentioning here that all recovery<br/> times are below 2 mins now.</p> <p>The error happened after executions of soak test for approximately ~ 73 hours.</p> <p>Sequence of events</p> <ul> <li>2016-03-18 00:34:30,557:fsmgmt.fsmgmt:INFO triggering fault mds_restart</li> <li>2016-03-18 00:40:49,655:fsmgmt.fsmgmt:INFO lola-8 is up!!</li> <li>2016-03-18 00:42:25,091:fsmgmt.fsmgmt:INFO ... soaked-MDT0000 mounted successfully on lola-8</li> <li>till then recovery process stalled and slab allocation is increasing continuously.</li> <li>2016-03-18 01:34 - 02:04 created debug logs with frequency ~ 4min</li> </ul> <p>The distribution of the biggest consumers is similar to the 2 MDTs per MDS configuration listed above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #Date Time SlabName ObjInUse ObjInUseB ObjAll ObjAllB SlabInUse SlabInUseB SlabAll SlabAllB SlabChg SlabPct size-1048576.dat:20160318 03:27:00 size-1048576 10115 10606346240 10115 10606346240 10115 10606346240 10115 10606346240 0 0 size-262144.dat:20160318 03:27:00 size-262144 449 117702656 449 117702656 449 117702656 449 117702656 0 0 size-8192.dat:20160318 03:27:00 size-8192 4663 38199296 4663 38199296 4663 38199296 4663 38199296 0 0 size-1024.dat:20160318 03:27:00 size-1024 35610 36464640 35628 36483072 8903 36466688 8907 36483072 106496 0 ptlrpc_cache.dat:20160318 03:27:00 ptlrpc_cache 41048 31524864 41080 31549440 8216 33652736 8216 33652736 53248 0 size-65536.dat:20160318 03:27:00 size-65536 360 23592960 360 23592960 360 23592960 360 23592960 0 0 size-512.dat:20160318 03:27:00 size-512 45384 23236608 45472 23281664 5684 23281664 5684 23281664 8192 0 kmem_cache.dat:20160318 03:27:00 kmem_cache 289 9506944 289 9506944 289 18939904 289 18939904 0 0 inode_cache.dat:20160318 03:27:00 inode_cache 15638 9257696 15684 9284928 2614 10706944 2614 10706944 0 0 Acpi-Operand.dat:20160318 03:27:00 Acpi-Operand 133270 9595440 135468 9753696 2556 10469376 2556 10469376 0 0 </pre> </div></div> <p>After the occurrance of the error, debug logs with filter '<tt>+malloc + trace</tt> have been taken for ~ 30 Mins in 4 minutes intervals. Buffer size was increased from initially 128MB to 1024 MB. <br/> Uploaded files:</p> <ul> <li>message, console and <tt>collectl</tt> slab performance counter lola-8</li> <li>I try to attache the kernel debug logs to the tickets or will upload them to a Intel cluster for further investigations if I'll face size limits or transfer problems.</li> </ul> <p>Also, it is impossible to abort the recovery process (see <a href="https://jira.hpdd.intel.com/browse/LU-7848?focusedCommentId=146077&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-146077" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-7848?focusedCommentId=146077&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-146077</a>)</p> <p>debug files have been uploaded. Oleg: I kept them in the binary state as I was unsure what should be extracted<br/> since Andreas requested '+trace' also.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19693" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19693</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7836" title="MDSes crashed with oom-killer" class="issue-link" data-issue-key="LU-7836"><del>LU-7836</del></a> ptlrpc: remove duplicate final ping req<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b6f85ba66403a30f135037ae38fb88b35d9b86aa</p> <p>oom-killer was active for build '20160427' (see <a href="https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160427" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/pages/viewpage.action?title=Soak+Testing+on+Lola&amp;spaceKey=Releases#SoakTestingonLola-20160427</a>) also. The patch above wasn't applied. Crash dump files are saved in:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lhn.hpdd.intel.com:/var/crashdumps/lu-7836/lola-11/127.0.0.1-2016-05-07-10:48:57 lhn.hpdd.intel.com:/var/crashdumps/lu-7836/lola-11/127.0.0.1-2016-05-07-17:34:05 </pre> </div></div> <p>The error also occurred while soak testing build '20160601' (see: <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160601" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160601</a>) after the recovery process stalled for a MDT that failed over to the secondary node. Although <a href="https://jira.whamcloud.com/browse/LU-7848" title="Recovery process on MDS stalled" class="issue-link" data-issue-key="LU-7848"><del>LU-7848</del></a> is part of the build, the recovery process is stalls.<br/> Configuration: DNE enabled; 1 MDT per MDS; nodes lola-10, 11 from an active-active failover cluster. </p> <p>1st Event:</p> <ul> <li>2016-06-03 11:31:10 - failover resource of lola-10 (MDT-2) --&gt; lola-11</li> <li>2016-06-03 11:36:37 - ... soaked-MDT0002 mounted successfully on lola-11</li> <li>till 2016-06-04-00:44 - soaked-MDT0002 in status 'RECOVERING'.</li> <li>2016-06-04-00:44:52 - lola-11 crash with oom-killer</li> </ul> <p>2nd Event:</p> <ul> <li>2016-06-07 08:34:06,621 triggering fault mds_failover lola-10 (MDT-2) --&gt; lola-11</li> <li>2016-06-07 08:38:42 - Mounting soaked-MDT0002 on lola-11</li> <li>since 2016-06-07 08:39:32,155 Wait for recovery to complete</li> <li>memory resources are nearly exhausted: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lola-11 ~]# date Wed Jun 8 07:59:49 PDT 2016 [root@lola-11 ~]# collectl -sm --verbose waiting for 1 second sample... # MEMORY SUMMARY #&lt;-------------------------------Physical Memory--------------------------------------&gt;&lt;-----------Swap------------&gt;&lt;-------Paging------&gt; # Total Used Free Buff Cached Slab Mapped Anon Commit Locked Inact Total Used Free In Out Fault MajFt In Out 32006M 30564M 1441M 127144K 676256K 28701M 16196K 69072K 201740K 5008K 509M 15G 0 15G 0 0 28 0 0 8 32006M 30565M 1441M 127144K 676256K 28701M 16196K 69072K 201740K 5008K 509M 15G 0 15G 0 0 63 0 0 4 32006M 30565M 1441M 127144K 676256K 28701M 16196K 69072K 201740K 5008K 509M 15G 0 15G 0 0 1 0 0 0 32006M 30564M 1441M 127144K 676256K 28701M 16196K 69072K 201740K 5008K 509M 15G 0 15G 0 0 17 0 0 0 </pre> </div></div> <p>Attached files:<br/> *1st event only: Saved crash dump file to <tt>lhn.hpdd.intel.com:/var/crashdumps/lu-7836/lola-11/127.0.0.1-2016-06-04-00:44:52</tt></p></li> <li>2nd event only: kernel debug log of lola-11</li> <li>Both event: messages, console logs</li> </ul> <p>I double checked the server node <tt>lola-11</tt> and found no HW related errors.</p> <p>The error didn't occurred for soak test of build <a href="https://build.hpdd.intel.com/job/lustre-master/3406" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-master/3406</a> (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713</a>) during a test session that is ongoing and last already for 7 days.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19693/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19693/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7836" title="MDSes crashed with oom-killer" class="issue-link" data-issue-key="LU-7836"><del>LU-7836</del></a> ptlrpc: remove duplicate final ping req<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1f2bb415543f8801f3b4a7c55e29b715b09f327b</p> <p>Is this issue resolved with the landing of the above patch?</p> <p>New build (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160727" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160727</a>) include the fix above has been started 1 hour ago. The previous test session for '20160713' ran till yesterday (Aug, 26th) without the occurrence of this bug.<br/> If possible I'd like to run build '20160727' till Friday before closing the ticket. Anyway feel free to close the ticket.</p> <p>Thanks Frank. We can wait until Friday to validate that the issue is resolved.</p> <p>Hi Frank,<br/> Are you comfortable that the issue is resolved in the soak run from the end of last week?<br/> Thanks.<br/> Joe</p> <p>As this fix has landed and is intended to fix this issue then let's mark this as resolved and then reopen if it reoccurs</p> Related LU-7780 LU-8070 Development Rank 1|hzy3db: Rank (Obsolete) 9223372036854775807 Severity