https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6607 Lustre <p>2 node DNE MDS<br/> 16 OSS<br/> 2K clients</p> <p>A MDS node randomly running out of memory and hang.<br/> We watch MDS drain its memory in matter of few minutes. Many times right after recovery from previous hangs. </p> <p>Clients are generating a ton of Lustre errors with strings "ptlrpc_expire_one_request". The numbers are from several hundred thousands to several millions of such errors from each node. Here are number of error counts from some nodes:</p> <p>comet-12-31 662616<br/> comet-10-06 690764<br/> comet-12-24 720396<br/> comet-12-25 735659<br/> comet-12-14 778073<br/> comet-12-33 840302<br/> comet-10-10 928322<br/> comet-12-33 945614<br/> comet-12-25 992288<br/> comet-10-15 1131711<br/> comet-12-25 1147043<br/> comet-10-07 1160876<br/> comet-12-30 1180270<br/> comet-10-03 1387072<br/> comet-10-02 2515764<br/> comet-10-02 3371128</p> <p>I am attaching logs from both client and server on one such incidence.</p> Linux panda-mds-19-6.sdsc.edu 3.10.73-1.el6.elrepo.x86_64 #1 SMP Thu Mar 26 16:28:30 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux<br/> <br/> lustre-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-osd-zfs-mount-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-iokit-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-source-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-osd-zfs-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-modules-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-tests-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> LU-6607

MDS ( 2 node DNE) running out of memory and crash

Bug Blocker Resolved Won't Fix Lai Siyao Haisong Cai sdsc Fri, 15 May 2015 16:06:25 +0000 Sat, 24 Mar 2018 14:03:17 +0000 Sat, 24 Mar 2018 14:03:17 +0000 Lustre 2.7.0 1 6 <p>Lai</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Just like to highlight these messages on server (should also be in messages-19-6.gz file)</p> <p>May 15 06:35:19 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 06:45:05 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:17:59 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:18:53 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:18:54 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:18:56 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:19:00 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:19:08 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:19:37 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:20:09 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:21:13 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:23:25 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:27:44 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 07:55:17 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:08:07 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:08:07 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:08:08 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:08:10 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:11:04 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:11:12 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). <br/> May 15 08:11:28 panda-mds-19-6 kernel: Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound). </p> <p>Hi Lai,</p> <p>Any update?</p> <p>thanks,<br/> Haisong</p> <p>Hello, Cai</p> <p>I checked the debug log and dmesg, and I can see MDT0001 seems very slow at that moment. though I can not figure out why from these message. So</p> <p>1. Could you please post these information here stack trace of MDT0001 (panda-mds-19-6), which will help us understand what MDT0001 was busying with. Something like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo t &gt; /proc/sysrq-trigger dmesg &gt; /tmp/dmesg.out </pre> </div></div> <p>2. Could you please post "cat /proc/slabinfo" here when OOM happens? </p> <p>Thanks<br/> WangDi </p> <p>Hi WangDi,</p> <p>I understand when to run 2). <br/> Do you want output of 1) now or at the same time when I run 2)?</p> <p>Haisong</p> <p>Hello, Cai</p> <p>Oh, I only need output of 1) when MDT1 is busy. But if you can get both at the same time, that would be great.</p> <p>Thanks<br/> WangDi</p> <p>WangDi,</p> <p>We had 2 incidences recently and both time I failed to collect need info.<br/> One time I simply forgot and the other time we had no chance since MDS node was hung.</p> <p>Haisong</p> <p>WangDi,</p> <p>We ran into this problem on one of MDS (mdt0, the master again today)<br/> I have collected information you asked by issuing the following commands:</p> <p>echo t &gt; /proc/sysrq-trigger<br/> dmesg &gt; /state/partition1/tmp/dmesg.out<br/> cat /proc/slabinfo &gt; /state/partition1/tmp/slabinfo.txt</p> <p>dmesg.out &amp; slabinfo.txt will be uploaded separately.</p> <p>Haisong</p> <p>Files collected between 2 time MDS crashes.</p> <p>Ah, it is a ZFS environment (ZFS + DNE)? A few questions here</p> <p>1. I saw this on your MDS console message(dmesg_mds.gz), the kernel version is definitely not EL6? EL7? But we do not support EL7 server on MDS yet. could you please confirm what kernel did you use on MDS?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Linux version 3.10.73-1.el6.elrepo.x86_64 (mockbuild@Build64R6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Thu Mar 26 16:28:30 EDT 2015 </pre> </div></div> <p>2. In the slab info</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kmalloc-8192 9033431 9033431 8192 1 2 : tunables 8 4 0 : slabdata 9033431 9033431 0 </pre> </div></div> <p>8192 size slab costs too much memory, 941G! that is too much. Btw: how much OSTs for each OSS?</p> <p>Hi WangDi,</p> <p>We are running CentOS 6.6 with Linux kernel 3.10.73 from elrepo.<br/> Lustre and ZFS are build as kdms modules.</p> <p>Filesystem has 16 OSS and each has 6 OSTs.</p> <p>Haisong</p> <p>On one of the 2 MDS servers:</p> <p><span class="error">&#91;root@panda-mds-19-6 panda-mds-19-6&#93;</span># sysctl -a | grep slab<br/> kernel.spl.kmem.slab_kmem_alloc = 92736<br/> kernel.spl.kmem.slab_kmem_max = 92736<br/> kernel.spl.kmem.slab_kmem_total = 172032<br/> kernel.spl.kmem.slab_vmem_alloc = 407675904<br/> kernel.spl.kmem.slab_vmem_max = 490480640<br/> kernel.spl.kmem.slab_vmem_total = 485459072<br/> vm.min_slab_ratio = 5</p> <p>Is that possible you can upgrade MDS to 2.7.58 ? there are quite a few fix on these area since 2.7.51.</p> <p>Btw: we are currently testing ZFS on DNE at <a href="https://jira.whamcloud.com/browse/LU-7009" title="Testing DNE on ZFS" class="issue-link" data-issue-key="LU-7009"><del>LU-7009</del></a>, please follow there.</p> <p><a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> is about OSS crashing problem. The OSS servers are part of these very same MDS servers. They are the one file-system.</p> <p>We are about to apply a new patch related to <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a>. It is built from <a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a></p> <p>Will it be satisfy your recommendation?</p> <p>Haisong</p> <p>Hmm, I think <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> is different issue. This ticket is about MDS OOM during failover? Do you happen to know any easy way to reproduce this problem?<br/> Hmm, btw: is that possible you can add "log_buf_len=10M" in your boot command? since the dmesg you post here only have half stack trace. Thanks.</p> <p>Hi Wang Di,</p> <p>I understand <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> is a different problem, for OSS and not MDS memory issue.</p> <p>What I said earlier was, to work on <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> problem, we have to apply a patch soon. This is because they are the same<br/> file-system. That patch is built with <a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a> </p> <p>Was that 2.7.58 equivalent?</p> <p>Haisong</p> <p>Ah, it is. you can use that build. Thanks</p> <p>Hi WangDi, </p> <p>You stated that 2.7.58 has a lot fixes. But it may still not fix our problem, correct? <br/> Can you elaborate on slab situation? You indicated 941G (or 94G) was too big, why is it? Is it because of default setting or some configuration mistake?</p> <p>thanks,<br/> Haisong</p> <p>Hello, Haisong</p> <p>Yes, I do not know the exact reason why for this 8192_size slab caused so much memory here. No, I do not think this is related with any default setting. Did you do a lot cross-MDT operation here, like creating remote directory or striped directory? (unfortunately, there are not enough stack trace information here) Btw: this stack trace is collected when OOM happens ? or before? or about to happen? Right now, I would suggest</p> <p>1. Use 2.7.58 plus that patch (<a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a>) you need, maybe also include <a href="http://review.whamcloud.com/#/c/16161/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16161/</a>. <br/> 2. Please add "log_bf_len=10M" in your boot command, so we can see more of the stack trace when error happens.<br/> 3. Please help me find an easy way to reproduce the problem. Thanks!</p> <p>Even though 2.7.58 might not help you on this issue, but it is way better than 2.7.51 on DNE. </p> <p>SDSC have moved onto more current releases so I do not think any further work is needed here</p> Development End date Fri, 4 Sep 2015 16:06:25 +0000 Rank 1|hzxdfb: Rank (Obsolete) 9223372036854775807 Severity Start date Fri, 15 May 2015 16:06:25 +0000