https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4195 Lustre <p>mdt response very slow. Top showed ptlrpcd running at 100% cpu. Console showed errors. Was able to run debug trace. See attached files.</p> <p>Lustre: Service thread pid 7065 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Pid: 7065, comm: mdt_93</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffff810539b2&gt;&#93;</span> ? enqueue_task_fair+0x142/0x490<br/> <span class="error">&#91;&lt;ffffffff81096f5d&gt;&#93;</span> ? sched_clock_cpu+0xcd/0x110<br/> <span class="error">&#91;&lt;ffffffffa04f960e&gt;&#93;</span> cfs_waitq_wait+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a9f6de&gt;&#93;</span> qos_statfs_update+0x7fe/0xa70 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8105fab0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa0aa00fd&gt;&#93;</span> alloc_qos+0x1ad/0x21a0 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0aa306c&gt;&#93;</span> qos_prep_create+0x1ec/0x2380 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04f98be&gt;&#93;</span> ? cfs_free+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a9c63a&gt;&#93;</span> lov_prep_create_set+0xea/0x390 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a84b0c&gt;&#93;</span> lov_create+0x1ac/0x1400 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d8a50f&gt;&#93;</span> ? obd_iocontrol+0xef/0x390 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d8f90e&gt;&#93;</span> mdd_lov_create+0x9ee/0x1ba0 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0da1871&gt;&#93;</span> mdd_create+0xf81/0x1a90 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ea9df3&gt;&#93;</span> ? osd_oi_lookup+0x83/0x110 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ea456c&gt;&#93;</span> ? osd_object_init+0xdc/0x3e0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0eda3f7&gt;&#93;</span> cml_create+0x97/0x250 <span class="error">&#91;cmm&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e165e1&gt;&#93;</span> ? mdt_version_get_save+0x91/0xd0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e2c06e&gt;&#93;</span> mdt_reint_open+0x1aae/0x28a0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa078f724&gt;&#93;</span> ? lustre_msg_add_version+0x74/0xd0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0da456e&gt;&#93;</span> ? md_ucred+0x1e/0x60 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e14c81&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e0bed4&gt;&#93;</span> mdt_reint_internal+0x544/0x8e0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e0c53d&gt;&#93;</span> mdt_intent_reint+0x1ed/0x530 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e0ac09&gt;&#93;</span> mdt_intent_policy+0x379/0x690 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074b351&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07711ad&gt;&#93;</span> ldlm_handle_enqueue0+0x48d/0xf50 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e0b586&gt;&#93;</span> mdt_enqueue+0x46/0x130 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e00772&gt;&#93;</span> mdt_handle_common+0x932/0x1750 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e01665&gt;&#93;</span> mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa079fb4e&gt;&#93;</span> ptlrpc_main+0xc4e/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa079ef00&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa079ef00&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa079ef00&gt;&#93;</span> ? ptlrpc_main+0x0/0x1a40 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> server running 2.1.5-2nas<br/> our source is at <a href="git://github.com/jlan/lustre-nas.git">git://github.com/jlan/lustre-nas.git</a> LU-4195

MDT Slow with ptlrpcd using 100% cpu.

Bug Major Resolved Cannot Reproduce Amir Shehata Mahmoud Hanafi Fri, 1 Nov 2013 18:01:13 +0000 Thu, 15 Sep 2016 22:11:56 +0000 Wed, 29 Oct 2014 16:30:59 +0000 Lustre 2.1.5 0 8 <p>Uploaded the following files to ftp site.</p> <p>LU4195.lustre-log.dump.selected.tgz<br/> LU4195.ptlrpcd.debug2.out.gz</p> <p>I also have crash dump that can be uploaded if needed.</p> <p>Lai</p> <p>What do you advise here?</p> <p>Peter</p> <p>Mahmoud, do you know what kind of application/operation caused the problem? I see there are quite lot of transaction commits in the log, looks like heavy load was on mds.</p> <p>Is it possible to get a full stack trace (especially the stack trace for ptlrpcd) by sysrq when this happened again? Thanks.</p> <p>Hope this helps</p> <p>crash&gt; bt 5560<br/> PID: 5560 TASK: ffff881fb5e87540 CPU: 8 COMMAND: "ptlrpcd"<br/> #0 <span class="error">&#91;ffff8820b0c07e90&#93;</span> crash_nmi_callback at ffffffff81029956<br/> #1 <span class="error">&#91;ffff8820b0c07ea0&#93;</span> notifier_call_chain at ffffffff815222f5<br/> #2 <span class="error">&#91;ffff8820b0c07ee0&#93;</span> atomic_notifier_call_chain at ffffffff8152235a<br/> #3 <span class="error">&#91;ffff8820b0c07ef0&#93;</span> notify_die at ffffffff81095f1e<br/> #4 <span class="error">&#91;ffff8820b0c07f20&#93;</span> do_nmi at ffffffff8151ff2f<br/> #5 <span class="error">&#91;ffff8820b0c07f50&#93;</span> nmi at ffffffff8151f7e0<br/> <span class="error">&#91;exception RIP: _spin_lock+33&#93;</span><br/> RIP: ffffffff8151f051 RSP: ffff881fac2b3bf0 RFLAGS: 00000297<br/> RAX: 0000000000000d28 RBX: 0000000000000004 RCX: 0000000000000000<br/> RDX: 0000000000000d27 RSI: 0000000000000007 RDI: ffffffffa05ada40<br/> RBP: ffff881fac2b3bf0 R8: 0000000000000246 R9: 0000000000000001<br/> R10: ffff883343e44cc0 R11: 0000000000000000 R12: 000500000a971b57<br/> R13: 0000000000003039 R14: 0000000000000000 R15: 000527015f9078c7<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff881fac2b3bf0&#93;</span> _spin_lock at ffffffff8151f051<br/> #7 <span class="error">&#91;ffff881fac2b3bf8&#93;</span> LNetMEAttach at ffffffffa057f99f <span class="error">&#91;lnet&#93;</span><br/> #8 <span class="error">&#91;ffff881fac2b3c68&#93;</span> ptl_send_rpc at ffffffffa078b532 <span class="error">&#91;ptlrpc&#93;</span><br/> #9 <span class="error">&#91;ffff881fac2b3d28&#93;</span> ptlrpc_send_new_req at ffffffffa0781993 <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff881fac2b3d98&#93;</span> ptlrpc_check_set at ffffffffa07840a8 <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff881fac2b3e38&#93;</span> ptlrpcd_check at ffffffffa07b4ed0 <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff881fac2b3e68&#93;</span> ptlrpcd at ffffffffa07b516e <span class="error">&#91;ptlrpc&#93;</span><br/> #13 <span class="error">&#91;ffff881fac2b3f48&#93;</span> kernel_thread at ffffffff8100c0ca</p> <p>crash&gt; bt 5560 -l<br/> PID: 5560 TASK: ffff881fb5e87540 CPU: 8 COMMAND: "ptlrpcd"<br/> #0 <span class="error">&#91;ffff8820b0c07e90&#93;</span> crash_nmi_callback at ffffffff81029956<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/arch/x86/include/asm/paravirt.h: 115<br/> #1 <span class="error">&#91;ffff8820b0c07ea0&#93;</span> notifier_call_chain at ffffffff815222f5<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/kernel/notifier.c: 95<br/> #2 <span class="error">&#91;ffff8820b0c07ee0&#93;</span> atomic_notifier_call_chain at ffffffff8152235a<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/kernel/notifier.c: 192<br/> #3 <span class="error">&#91;ffff8820b0c07ef0&#93;</span> notify_die at ffffffff81095f1e<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/kernel/notifier.c: 573<br/> #4 <span class="error">&#91;ffff8820b0c07f20&#93;</span> do_nmi at ffffffff8151ff2f<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/arch/x86/kernel/traps.c: 513<br/> #5 <span class="error">&#91;ffff8820b0c07f50&#93;</span> nmi at ffffffff8151f7e0<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/arch/x86_64/kernel/entry.S<br/> <span class="error">&#91;exception RIP: _spin_lock+33&#93;</span><br/> RIP: ffffffff8151f051 RSP: ffff881fac2b3bf0 RFLAGS: 00000297<br/> RAX: 0000000000000d28 RBX: 0000000000000004 RCX: 0000000000000000<br/> RDX: 0000000000000d27 RSI: 0000000000000007 RDI: ffffffffa05ada40<br/> RBP: ffff881fac2b3bf0 R8: 0000000000000246 R9: 0000000000000001<br/> R10: ffff883343e44cc0 R11: 0000000000000000 R12: 000500000a971b57<br/> R13: 0000000000003039 R14: 0000000000000000 R15: 000527015f9078c7<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff881fac2b3bf0&#93;</span> _spin_lock at ffffffff8151f051<br/> /usr/src/debug/kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/arch/x86/include/asm/spinlock.h: 127<br/> #7 <span class="error">&#91;ffff881fac2b3bf8&#93;</span> LNetMEAttach at ffffffffa057f99f <span class="error">&#91;lnet&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lnet/lnet/lib-me.c: 158<br/> #8 <span class="error">&#91;ffff881fac2b3c68&#93;</span> ptl_send_rpc at ffffffffa078b532 <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lustre/ptlrpc/niobuf.c: 614<br/> #9 <span class="error">&#91;ffff881fac2b3d28&#93;</span> ptlrpc_send_new_req at ffffffffa0781993 <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lustre/ptlrpc/client.c: 1437<br/> #10 <span class="error">&#91;ffff881fac2b3d98&#93;</span> ptlrpc_check_set at ffffffffa07840a8 <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lustre/ptlrpc/client.c: 1468<br/> #11 <span class="error">&#91;ffff881fac2b3e38&#93;</span> ptlrpcd_check at ffffffffa07b4ed0 <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lustre/ptlrpc/ptlrpcd.c: 240<br/> #12 <span class="error">&#91;ffff881fac2b3e68&#93;</span> ptlrpcd at ffffffffa07b516e <span class="error">&#91;ptlrpc&#93;</span><br/> /usr/src/redhat/BUILD/lustre-2.1.5/lustre/ptlrpc/ptlrpcd.c: 326<br/> #13 <span class="error">&#91;ffff881fac2b3f48&#93;</span> kernel_thread at ffffffff8100c0ca<br/> /usr/src/debug////////kernel-lustre215-2.6.32-279.19.1.el6/linux-2.6.32-279.19.1.el6.x86_64/arch/x86/kernel/entry_64.S: 1213</p> <p>Could you provide the full stack trace (for all tasks on all CPUs) as well? What about the memmory usage? Can this situation be recovered?</p> <p>In debug logs there are lots of messages like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:02000400:2.0:1383201992.510338:0:7620:0:(lib-move.c:1454:lnet_send()) No route to 12345-10.153.1.199@o2ib233 via 10.151.27.60@o2ib (all routers down) </pre> </div></div> <p>It looks like routers are down, and all connections are down too, and clients kept reconnecting to MDS (but failed), so ptlrpcd are always 100% busy.</p> <p>We were not able to recover from this and had to dump the system.<br/> See attached files for memory and stack track info</p> <p>We have 7 other filesystem that use the same routers and clients and they were not experiencing this issue. </p> <p>There is "some" evidence that this may have been triggered by running a "lfs setquota" command.</p> <p>I agree with Lai, the log shows that no router for o2ib to o2ib233, and the stack trace shows all ptlrpcd threads were busy on acquiring LNET_LOCK (which I think is caused by the router problem).</p> <p>Mahmoud, I didn't see anything related to quota in the log, what kind of evidence indicating it's triggered by 'lfs setquota' command?</p> <p>Amir,<br/> Could you take a look at this one?<br/> Thank you!</p> <p>Would it be possible to grab the route configuration for one of the nodes that have the problem? Also if you could please highlight the problematic routes.</p> <p>As basic sanity check, please make sure that routers are actually configured as routers, IE: forwarding="enabled". If not, then the node will drop all messages not destined to itself.</p> <p>This error message is hit whenever LNET tries to send a message to a final destination that exists on a net that do not have a route from the net the current node is on. Thus an appropriate route is chosen.</p> <p>Furthermore, the routes can exist but they might be down because the NID specified is not reachable. Is it possible to try and do an <b>lctl ping &lt;nid&gt;</b> from the node that's having the problem. That should return a list of NIDs of the target router and their status (up/down). If one of these nids are down and <b>avoid_asym_router_failure</b> is 1 (which it is by default), then the entire router is considered down and when sending messages we'll hit the above error.</p> <p>NOTE: if you have a router that has multiple nids, but one of the nids is "unused" (IE, it sends/receives no messages), this would cause that NID to be considered down, and will lead to the above described scenario.</p> <p>Hello Mahmoud,<br/> Just want to check in to see if this is still a 'live' issue for you?<br/> If the problem has been dealt with, can we mark this issue as 'resolved'?<br/> Thanks,<br/> ~ jfc.</p> <p>Hello again Mahmoud,<br/> Please advise if you want us to keep this ticket open?</p> <p>If we don't hear back we'll mark it as resolved, with no fix, and we can re-open it if requested to do that.</p> <p>Thanks,<br/> ~ jfc.</p> <p>please close.</p> <p>ok thanks Mahmoud</p> Development Epic hang server Rank 1|hzw7jr: Rank (Obsolete) 11369 Severity