https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1663 Lustre <p>At NOAA, there are two filesystems that were installed at the same time, lfs1 and lfs2. Recently lfs2 has been having MDS lockups, which cause a failover to the second MDS. It seems to run ok for a couple days and then whichever MDS is currently running will lockup and failover to the other one. lfs1, however, is not affected, though it runs an identical set up as far as hardware goes. </p> <p>We have the stack traces that get logged, but not the lustre-logs, as they have been on tmpfs. We've changed the debug_file location, so hopefully we'll get the next batch. I'll put a sampling of the interesting call traces, and attach the rest.</p> <p>Here is the root cause of the failover. The health_check times out and prints NOT HEALTHY, which causes ha to failover:<br/> Jul 17 17:23:30 lfs-mds-2-2 kernel: LustreError: 16021:0:(service.c:2124:ptlrpc_service_health_check()) mds: unhealthy - request has been waiting 725s</p> <p>This one makes it look like it might be quota related:<br/> Jul 17 17:14:04 lfs-mds-2-2 kernel: Call Trace:<br/> Jul 17 17:14:04 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff887f9220&gt;&#93;</span> :lnet:LNetPut+0x730/0x840<br/> Jul 17 17:14:04 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff800649fb&gt;&#93;</span> __down+0xc3/0xd8<br/> Jul 17 17:14:04 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff8008e421&gt;&#93;</span> default_wake_function+0x0/0xe<br/> Jul 17 17:14:04 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff88a29490&gt;&#93;</span> :lquota:dqacq_handler+0x0/0xc20<br/> ...</p> <p>This one looks a little like <a href="https://jira.whamcloud.com/browse/LU-1395" title="MDS hangs after calltrace at ldlm_expired_completion_wait()" class="issue-link" data-issue-key="LU-1395"><del>LU-1395</del></a> or <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a>:<br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: Call Trace:<br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888ceb51&gt;&#93;</span> ldlm_resource_add_lock+0xb1/0x180 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888e2a00&gt;&#93;</span> ldlm_expired_completion_wait+0x0/0x250 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff8006388b&gt;&#93;</span> schedule_timeout+0x8a/0xad<br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff8009987d&gt;&#93;</span> process_timeout+0x0/0x5<br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888e4555&gt;&#93;</span> ldlm_completion_ast+0x4d5/0x880 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888c9709&gt;&#93;</span> ldlm_lock_enqueue+0x9d9/0xb20 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff8008e421&gt;&#93;</span> default_wake_function+0x0/0xe<br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888c4b6a&gt;&#93;</span> ldlm_lock_addref_internal_nolock+0x3a/0x90 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888e30bb&gt;&#93;</span> ldlm_cli_enqueue_local+0x46b/0x520 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff88c611a7&gt;&#93;</span> enqueue_ordered_locks+0x387/0x4d0 <span class="error">&#91;mds&#93;</span><br/> Jul 4 17:58:29 lfs-mds-2-2 kernel: <span class="error">&#91;&lt;ffffffff888e09a0&gt;&#93;</span> ldlm_blocking_ast+0x0/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> ...</p> Lustre 1.8.6.80, jenkins-g9d9d86f-PRISTINE-2.6.18-238.12.1.el5_lustre.gd70e443<br/> Centos 5.5<br/> LU-1663

MDS threads hang for over 725s, causing fail over

Bug Major Resolved Fixed Oleg Drokin Kit Westneat mn8 patch Mon, 23 Jul 2012 20:31:04 +0000 Fri, 4 Oct 2013 15:33:47 +0000 Fri, 4 Oct 2013 15:33:37 +0000 Lustre 1.8.7 0 7 <p>grepped out call traces</p> <p>mds1 log</p> <p>mds2 log</p> <p>Oleg is already looking into <a href="https://jira.whamcloud.com/browse/LU-1395" title="MDS hangs after calltrace at ldlm_expired_completion_wait()" class="issue-link" data-issue-key="LU-1395"><del>LU-1395</del></a></p> <p>Kit, in your logs you only show mds side.<br/> the ast waiting traces (loike the second one) imply that the client did not cancel a blocking lock. We need to get client side log to see why.<br/> Preferably client side should have dlmtrace log level enabled</p> <p>Oleg, I will try to get this from the customer for the next failover event. Are there any additional logs from the MDS that might be useful?</p> <p>Well, getting a matching dlmtrace-enabled log from MDS at the same time would be helpful too</p> <p>logs from the client</p> <p>MDS logs</p> <p>These are logs from the last time the failover happened, before the debugging got turned on. It looks like many of the clients are dumping stacks on close operations:</p> <p>/var/log/messages-20120722:Jul 21 12:22:25 r32i3n11 kernel: <span class="error">&#91;&lt;ffffffffa06dbf25&gt;&#93;</span> ll_close_inode_openhandle+0x175/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> /var/log/messages-20120722:Jul 21 12:22:25 r32i3n11 kernel: <span class="error">&#91;&lt;ffffffffa06dc549&gt;&#93;</span> ll_mdc_real_close+0xf9/0x340 <span class="error">&#91;lustre&#93;</span><br/> /var/log/messages-20120722:Jul 21 12:22:25 r32i3n11 kernel: <span class="error">&#91;&lt;ffffffffa06ec4a2&gt;&#93;</span> ll_mdc_close+0x222/0x3f0 [lustre</p> <p>/var/log/messages-20120722:Jul 21 12:21:21 r32i3n5 kernel: <span class="error">&#91;&lt;ffffffffa06dbf25&gt;&#93;</span> ll_close_inode_openhandle+0x175/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> /var/log/messages-20120722:Jul 21 12:21:21 r32i3n5 kernel: <span class="error">&#91;&lt;ffffffffa06dc549&gt;&#93;</span> ll_mdc_real_close+0xf9/0x340 <span class="error">&#91;lustre&#93;</span><br/> /var/log/messages-20120722:Jul 21 12:21:21 r32i3n5 kernel: <span class="error">&#91;&lt;ffffffffa06ec4a2&gt;&#93;</span> ll_mdc_close+0x222/0x3f0 <span class="error">&#91;lustre&#93;</span><br/> /var/log/messages-20120722:Jul 21 12:21:21 r32i3n5 kernel: <span class="error">&#91;&lt;ffffffffa07124db&gt;&#93;</span> ? ll_stats_ops_tally+0x6b/0xd0 [lustre</p> <p>etc.</p> <p>It happened again, but we were unable to get the debug logs. I've modified the failover script to kernel panic the MDS, so hopefully we'll get a core dump. </p> <p>Is there anything useful in the logs I uploaded before? I can upload the logs we did get from this last event, but it looks like more of the same.</p> <p>NOAA has had to turn off the dlmtrace debug, as it causes too big of a performance hit. I'm still trying to get lctl dk from one of the events though.</p> <p>Can I get an opinion on the logs that are uploaded? Any thing interesting in them at all?</p> <p>Also, this bug looks similar in terms of symptoms, though it seems to be more directly related to unlinks than to closes:<br/> <a href="https://bugzilla.clusterstor.com/show_bug.cgi?id=699&amp;list_id=cookie" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.clusterstor.com/show_bug.cgi?id=699&amp;list_id=cookie</a></p> <p>The last set of logs looks to be pretty different from first set of logs.<br/> Looks like some sort of journal/quota deadlock.</p> <p>Kit</p> <p>Would it be possible to collect a crashdump the next time that this occurs?</p> <p>Thanks</p> <p>Peter</p> <p>We've been trying to get stack traces and crashdumps, but for one reason or another it hasn't worked. We did find a lustre-log recently that may or may not be related. I will upload it anyway. We also recorded some information after the last failover, but without the stack trace or crashdump, I'm afraid it's useless. I will upload it anyway, maybe you all can suggest some other information to get. </p> <p>lctl dk from certain clients, logs from MDS (cut off by premature forced kernel panic)</p> <p>lustre log from 9/10</p> <p>The customer is getting anxious about this issue. We are testing our kdump configuration now to ensure that we have a crashdump, but can you please review the logs and see if there is any more information we should gather? The next hang needs to be the last. Thanks, Kit. </p> <p>kern.log from most recent failover (with sysrq-t)</p> <p>Hi we were finally able to get some good debugging data from the system during a failover. Here is the sysrq-t from before it went down. I also managed to save off all the lustre-logs, there are a ton as you can see in the kern.log. Let me know if you want to see them.</p> <p>I noticed that there was a slab reclaim going on.. Wasn't there another issue with slab reclaims taking a long time?</p> <p>Any update?</p> <p>Thanks,<br/> Kit</p> <p>Hm, I do not see any obvious culprits in the traces.<br/> A bit strange how jbd thread is the first to complain it's hung and in the middle of transaction commit with no report of any sleep, yet in D state (I wish you had a crashdump so that it could be checked more closely).<br/> Anyway the first log that should hold most of the data is /tmp/lustre-log.1361661252.1181 I guess? (the biggest one around that timestamp is the one we need) and I guess we need it to at least try to see what was going on at the time.</p> <p>Also I see an umount in the logs, but I assume that's after the system was wedged for some time and your scripts just attempt it?</p> <p>Hi Oleg,</p> <p>That's correct, the system tried to umount once the OSTs went unhealthy, but failed.</p> <p>The lustre-log looks pretty slim, hopefully you can find something I missed.</p> <p>I will try to get a crashdump. Unfortunately all the ones we've been able to get so far have been incomplete. I'll try to come up with a way to get a full crashdump. Are there any other logs/debug information we can get that won't have too much impact on the running system?</p> <p>Thanks,<br/> Kit </p> <p>Getting the full crashdump from a wedged transaction commit like this should be most useful I suspect.</p> <p>I suspect what you are seeing is a generic jbd2 transaction wraparound problem.<br/> There were several rounds at RedHat to fix this, previous round here (already part of rhel6.4 kernel, no idea if they backported to rhel5 too): <a href="https://bugzilla.redhat.com/show_bug.cgi?id=735768" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=735768</a> and then the latest one seems to be unfolding here: <a href="http://lists.debian.org/debian-kernel/2013/03/msg00313.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.debian.org/debian-kernel/2013/03/msg00313.html</a> though I don't know when it will make it to rhel kernels either.</p> <p>I don't see that bug fixed in any RHEL5 kernel. Since Redhat no longer breaks out their patches, it seems like it would be difficult to figure out what they did to fix it. I will look at the SRPMS for the jbd stuff to see if I can figure it out. Have you seen the transaction wraparound problem in Lustre before? Are there any workarounds? It seems like we can't be the first to hit it.</p> <p>I have a high-load test system for lustre regression testing and I am hitting two sorts of jbd issues somewhat frequently, the rh bz735768 and then some jbd thread lockups that I think are related. Some of them are somewhat similar to what you see, I think. So I was just doing some more investigations into that and uncovered that they are apparently known problems if you have high transaction rates (which I certainly do).</p> <p>I suspect you can cherry-pick the patches from mainline linux kernel.</p> <p>It looks like the patch RH took to solve bz735768 is this one:<br/> <a href="https://github.com/torvalds/linux/commit/deeeaf13b291420fe4a4a52606b9fc9128387340" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/torvalds/linux/commit/deeeaf13b291420fe4a4a52606b9fc9128387340</a></p> <p>Oh I just realized that the debian-kernel patch references the two commits already, doh. </p> <p>Are there any plans to carry the debian-kernel patch until RH backports it? How risky do you think it is to backport them to RHEL5?</p> <p>We finally got a full vmcore. It's 12GB.. Is there a good place to put it? Alternatively I could run commands on it. What would be the best way to prove that it's the tid wraparound issue?</p> <p>Thanks.</p> <p>Backtrace:<br/> PID: 23917 TASK: ffff810abe18c7e0 CPU: 2 COMMAND: "vgck"<br/> #0 <span class="error">&#91;ffff810b35bd75a0&#93;</span> crash_kexec at ffffffff800b1192<br/> #1 <span class="error">&#91;ffff810b35bd7660&#93;</span> __die at ffffffff80065137<br/> #2 <span class="error">&#91;ffff810b35bd76a0&#93;</span> die at ffffffff8006c735<br/> #3 <span class="error">&#91;ffff810b35bd76d0&#93;</span> do_invalid_op at ffffffff8006ccf5<br/> #4 <span class="error">&#91;ffff810b35bd7790&#93;</span> error_exit at ffffffff8005ddf9<br/> <span class="error">&#91;exception RIP: jbd2_journal_start+58&#93;</span><br/> RIP: ffffffff8b32ae31 RSP: ffff810b35bd7848 RFLAGS: 00010206<br/> RAX: ffff810c1aef19c0 RBX: ffff81061476dd38 RCX: ffff810910fdcc00<br/> RDX: ffffffffffffffe2 RSI: 0000000000000012 RDI: ffff81061476dd38<br/> RBP: ffff810919769800 R8: ffff81000005b600 R9: ffff8104cd371a80<br/> R10: 0000000000000000 R11: ffffffff8b36cb60 R12: 0000000000000012<br/> R13: ffff810b35bd78c8 R14: 0000000000000080 R15: 0000000000000100<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> #5 <span class="error">&#91;ffff810b35bd7860&#93;</span> ldiskfs_dquot_drop at ffffffff8b36fdbe<br/> #6 <span class="error">&#91;ffff810b35bd7880&#93;</span> clear_inode at ffffffff800234e7<br/> #7 <span class="error">&#91;ffff810b35bd7890&#93;</span> dispose_list at ffffffff800352d6<br/> #8 <span class="error">&#91;ffff810b35bd78c0&#93;</span> shrink_icache_memory at ffffffff8002dcfb<br/> #9 <span class="error">&#91;ffff810b35bd7900&#93;</span> shrink_slab at ffffffff8003f7cd<br/> #10 <span class="error">&#91;ffff810b35bd7940&#93;</span> zone_reclaim at ffffffff800cf9ae<br/> #11 <span class="error">&#91;ffff810b35bd79f0&#93;</span> get_page_from_freelist at ffffffff8000a8a7<br/> #12 <span class="error">&#91;ffff810b35bd7a60&#93;</span> __alloc_pages at ffffffff8000f48a<br/> #13 <span class="error">&#91;ffff810b35bd7ad0&#93;</span> cache_grow at ffffffff80017a52<br/> #14 <span class="error">&#91;ffff810b35bd7b20&#93;</span> cache_alloc_refill at ffffffff8005c3ee<br/> #15 <span class="error">&#91;ffff810b35bd7b60&#93;</span> kmem_cache_alloc at ffffffff8000ac96<br/> #16 <span class="error">&#91;ffff810b35bd7b80&#93;</span> mb_cache_entry_alloc at ffffffff801063df<br/> #17 <span class="error">&#91;ffff810b35bd7ba0&#93;</span> ext3_xattr_cache_insert at ffffffff8805bb1e<br/> #18 <span class="error">&#91;ffff810b35bd7bd0&#93;</span> ext3_xattr_get at ffffffff8805cd3a<br/> #19 <span class="error">&#91;ffff810b35bd7c50&#93;</span> ext3_get_acl at ffffffff8805d4e2<br/> #20 <span class="error">&#91;ffff810b35bd7ca0&#93;</span> ext3_init_acl at ffffffff8805d8de<br/> #21 <span class="error">&#91;ffff810b35bd7ce0&#93;</span> ext3_new_inode at ffffffff8804e21f<br/> #22 <span class="error">&#91;ffff810b35bd7db0&#93;</span> ext3_create at ffffffff88054b6b<br/> #23 <span class="error">&#91;ffff810b35bd7e00&#93;</span> vfs_create at ffffffff8003aa1b<br/> #24 <span class="error">&#91;ffff810b35bd7e30&#93;</span> open_namei at ffffffff8001b749<br/> #25 <span class="error">&#91;ffff810b35bd7ea0&#93;</span> do_filp_open at ffffffff80027ba1<br/> #26 <span class="error">&#91;ffff810b35bd7f50&#93;</span> do_sys_open at ffffffff80019fa5<br/> #27 <span class="error">&#91;ffff810b35bd7f80&#93;</span> tracesys at ffffffff8005d29e (via system_call)<br/> RIP: 0000003d598c5b40 RSP: 00007fff8f6ae3b8 RFLAGS: 00000246<br/> RAX: ffffffffffffffda RBX: ffffffff8005d29e RCX: ffffffffffffffff<br/> RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 00007fff8f6ae490<br/> RBP: 00007fff8f6ae4e0 R8: 0000000000000004 R9: 0000000000000000<br/> R10: 0000000000000241 R11: 0000000000000246 R12: 000000000e3a5e90<br/> R13: 0000000000484f87 R14: 0000000000000004 R15: 000000000e3a5e90<br/> ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b</p> <p>Dennis pointed out that it looks like <a href="https://jira.whamcloud.com/browse/LU-3071" title="kernel BUG at fs/jbd2/transaction.c:293 (jbd2_journal_start)" class="issue-link" data-issue-key="LU-3071"><del>LU-3071</del></a>. Reading that bug, it looks like it should affect MDTs and 1.8 as well, is that correct? The assertion hit is at the same place:<br/> J_ASSERT(handle-&gt;h_transaction-&gt;t_journal == journal);</p> <p>yes, this latest trace is a different issue similar to lu-3071, but in your trace there is no lustre in the picture at all.<br/> It sounds there's a bug in mballoc in your ext3 code that allocates mb_cache entry with a wrong flag, should be GFP_NOFS.</p> <p>I checked in rhel5 kernel I have around and the bug is there in fs/mbcache.c:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct mb_cache_entry * mb_cache_entry_alloc(struct mb_cache *cache) { struct mb_cache_entry *ce; atomic_inc(&amp;cache-&gt;c_entry_count); ce = kmem_cache_alloc(cache-&gt;c_entry_cache, GFP_KERNEL); &lt;===== BUG </pre> </div></div> <p>In rhel6 I see they actually pass a gfp flags to this function to be used by the alloc to avoid this, you might be able to hunt down the patch that fixes that I guess.</p> <p>as for the vmcore, put it somewhere where we can get it? we also would need kernel-debuginfo you used at the very least. lustre modules might be useful just in case too.</p> <p>Does this patch look correct?</p> <p><a href="http://git.openvz.org/?p=linux-2.6.32-openvz;a=commitdiff;h=335e92e8a515420bd47a6b0f01cb9a206c0ed6e4" class="external-link" target="_blank" rel="nofollow noopener">http://git.openvz.org/?p=linux-2.6.32-openvz;a=commitdiff;h=335e92e8a515420bd47a6b0f01cb9a206c0ed6e4</a></p> <p>We are just using the default 1.8.9 kernel:<br/> <a href="http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/el5/server/RPMS/x86_64/kernel-debuginfo-2.6.18-348.1.1.el5_lustre.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/el5/server/RPMS/x86_64/kernel-debuginfo-2.6.18-348.1.1.el5_lustre.x86_64.rpm</a></p> <p>Dennis is uploading the coredump now and will update the ticket when it is finished. </p> <p>I pushed the patch we are using for the transaction wrap around in case it would be useful:<br/> <a href="http://review.whamcloud.com/6147" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6147</a></p> <p>Ok, I got vmcore, but the vmlinux from openvz is not it.</p> <p>In any case I looked inside of the core and it looks like it's for this latest crash that we already know what happened that is a different bug altogether.</p> <p>What I am looking for is vmcore from a situation where jbd2 commit threads locks up.</p> <p>The openviz link was just supposed to be a link to the patch, the kernel we are running is a stock Centos kernel. The backtrace and the core come from the same crash. There is another core, but it was placed on the wrong disk and we need to wait for a downtime to get it (next Tuesday). If it is a different backtrace, I'll update the ticket then. </p> <p>As far as the fix for the GFP_NOFS bug, does that patch work? I found the patch on git.kernel.org as well:<br/> <a href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=335e92e8a515420bd47a6b0f01cb9a206c0ed6e4" class="external-link" target="_blank" rel="nofollow noopener">http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=335e92e8a515420bd47a6b0f01cb9a206c0ed6e4</a></p> <p>Should I backport that to b1_8/RHEL5?</p> <p>Thanks.</p> <p>yes, this patch looks like it would do the right thing.</p> <p>Hi, would you please review this patch sonner? we are hitting multple server crash due to this issue.</p> <p>ok Ihara. Patch is at <a href="http://review.whamcloud.com/#/c/6147/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6147/</a></p> <p>Hi Peter, yes, but, we are waiting inspection. Once code review is done, we will apply this to the kernel for servers.</p> <p>Ihara </p> <p>Sorry that my comment was not clear enough. I understand that you wish to have reviews on the patch and I was acknowledging that and then separately adding the reference to the patch. It is required to include such a link in the JIRA ticket to cross-reference between JIRA and gerrit and this step had previously been overlooked</p> <p>Peter</p> <p>I also just submitted the xattrs patch that was also referenced. We are carrying this patch already at NOAA and it seems to improve stability. </p> <p><a href="http://review.whamcloud.com/7788" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7788</a></p> <p>RH patches ok to use in production</p> Development Rank 1|hzv3fj: Rank (Obsolete) 4055 Severity