https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1311 Lustre <p>ORNL reported this lockup</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 446.217881] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2 [ 446.217881] Pid: 9709, comm: kiblnd_sd_05 Not tainted 2.6.32-220.el6.wc.x86_64 #1 [ 446.217881] Call Trace: [ 446.217881] &lt;NMI&gt; [&lt;ffffffff814ec681&gt;] ? panic+0x78/0x143 [ 446.217881] [&lt;ffffffff810d8fad&gt;] ? watchdog_overflow_callback+0xcd/0xd0 [ 446.217881] [&lt;ffffffff8110a89d&gt;] ? __perf_event_overflow+0x9d/0x230 [ 446.217881] [&lt;ffffffff8110ae54&gt;] ? perf_event_overflow+0x14/0x20 [ 446.217881] [&lt;ffffffff8101e096&gt;] ? intel_pmu_handle_irq+0x336/0x550 [ 446.217881] [&lt;ffffffff814f2256&gt;] ? kprobe_exceptions_notify+0x16/0x430 [ 446.217881] [&lt;ffffffff814f0d39&gt;] ? perf_event_nmi_handler+0x39/0xb0 [ 446.217881] [&lt;ffffffff814f2885&gt;] ? notifier_call_chain+0x55/0x80 [ 446.217881] [&lt;ffffffff814f28ea&gt;] ? atomic_notifier_call_chain+0x1a/0x20 [ 446.217881] [&lt;ffffffff81096bce&gt;] ? notify_die+0x2e/0x30 [ 446.217881] [&lt;ffffffff814f0503&gt;] ? do_nmi+0x173/0x2b0 [ 446.217881] [&lt;ffffffff814efe10&gt;] ? nmi+0x20/0x30 [ 446.217881] [&lt;ffffffff814ef67c&gt;] ? _spin_lock+0x1c/0x30 [ 446.217881] &lt;&lt;EOE&gt;&gt; [&lt;ffffffffa07473bf&gt;] ? cfs_trace_lock_tcd+0x2f/0x90 [libcfs] [ 446.217881] [&lt;ffffffffa0750b0a&gt;] ? libcfs_debug_vmsg2+0xda/0xb60 [libcfs] [ 446.217881] [&lt;ffffffffa07515d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 446.217881] [&lt;ffffffffa07515d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 446.217881] [&lt;ffffffffa0b5768d&gt;] ? kiblnd_scheduler+0x16d/0x620 [ko2iblnd] [ 446.217881] [&lt;ffffffff8105fa50&gt;] ? default_wake_function+0x0/0x20 [ 446.217881] [&lt;ffffffffa0b57520&gt;] ? kiblnd_scheduler+0x0/0x620 [ko2iblnd] [ 446.217881] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [ 446.217881] [&lt;ffffffffa0b57520&gt;] ? kiblnd_scheduler+0x0/0x620 [ko2iblnd] [ 446.217881] [&lt;ffffffffa0b57520&gt;] ? kiblnd_scheduler+0x0/0x620 [ko2iblnd] [ 446.217881] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>the other thread is in irq context:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 9855 TASK: ffff880431898b40 CPU: 0 COMMAND: <span class="code-quote">"ktracefiled"</span> #0 [ffff880028207e90] crash_nmi_callback at ffffffff81029716 #1 [ffff880028207ea0] notifier_call_chain at ffffffff814f2885 #2 [ffff880028207ee0] atomic_notifier_call_chain at ffffffff814f28ea #3 [ffff880028207ef0] notify_die at ffffffff81096bce #4 [ffff880028207f20] do_nmi at ffffffff814f0503 #5 [ffff880028207f50] nmi at ffffffff814efe10 [exception RIP: _spin_lock_irqsave+47] RIP: ffffffff814ef56f RSP: ffff880028203de8 RFLAGS: 00000097 RAX: 00000000000055c2 RBX: ffff88040df2b800 RCX: 00000000000055c1 RDX: 0000000000000046 RSI: ffff88040df2b800 RDI: ffffffffa0b71c30 RBP: ffff880028203de8 R8: ffff8803fbb04000 R9: ffff8803fbb05de0 R10: ffff880028403a40 R11: 0000000000000000 R12: 0000000000000087 R13: 0000000000000000 R14: ffff8804318da000 R15: ffff8803fbb05f00 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #6 [ffff880028203de8] _spin_lock_irqsave at ffffffff814ef56f #7 [ffff880028203df0] kiblnd_cq_completion at ffffffffa0b51548 [ko2iblnd] &lt;== <span class="code-keyword">this</span> is line 3277 (cfs_spin_lock_irqsave(&amp;kiblnd_data.kib_sched_lock, flags);) in o2iblnd_cb.c #8 [ffff880028203e10] mthca_cq_completion at ffffffffa019ca4a [ib_mthca] #9 [ffff880028203e30] mthca_eq_int at ffffffffa019c422 [ib_mthca] #10 [ffff880028203eb0] mthca_arbel_msi_x_interrupt at ffffffffa019c694 [ib_mthca] #11 [ffff880028203ed0] handle_IRQ_event at ffffffff810d94f0 #12 [ffff880028203f20] handle_edge_irq at ffffffff810dbc2e #13 [ffff880028203f60] handle_irq at ffffffff8100df09 #14 [ffff880028203f80] do_IRQ at ffffffff814f504c --- &lt;IRQ stack&gt; --- #15 [ffff8803fbb05d38] ret_from_intr at ffffffff8100ba53 [exception RIP: put_pages_on_tcd_daemon_list+255] RIP: ffffffffa075072f RSP: ffff8803fbb05de0 RFLAGS: 00000246 RAX: ffffea000c9a82a8 RBX: ffff8803fbb05e20 RCX: ffff880390407b88 RDX: ffff88040d080128 RSI: ffff880390407b48 RDI: ffff880390407b68 RBP: ffffffff8100ba4e R8: ffff88040d080128 R9: 0000000000000000 R10: ffff880028403a40 R11: 0000000000000000 R12: ffff8803fbb05d70 R13: ffff8803fbb05f00 R14: ffff8803fbb05ef0 R15: 0000000000000286 ORIG_RAX: ffffffffffffff46 CS: 0010 SS: 0018 #16 [ffff8803fbb05e28] put_pages_on_daemon_list at ffffffffa075094b [libcfs] &lt;== <span class="code-keyword">this</span> is line 575 (cfs_tcd_for_each_type_lock) in tracefile.c #17 [ffff8803fbb05e78] tracefiled at ffffffffa0751d4f [libcfs] #18 [ffff8803fbb05f48] kernel_thread at ffffffff8100c14a </pre> </div></div> LU-1311

Debugging deadlock in lnet

Bug Blocker Resolved Fixed Oleg Drokin Oleg Drokin Thu, 12 Apr 2012 11:12:55 +0000 Thu, 31 May 2012 16:36:36 +0000 Thu, 31 May 2012 16:36:36 +0000 Lustre 2.2.0 Lustre 2.3.0 Lustre 2.3.0 0 3 <p>tracefiled thread: lock some random tcd (nonirq type)<br/> ibscheduler thread: get kiblnd_data.kib_sched_lock<br/> ibscheduler thread: try to print debug message, need the lock in the tcd that tracefiled holds<br/> tracefiled: interrupt hits, needs the kiblnd_data.kib_sched_lock == deadlock</p> <p>Oleg, due to your analysis, any call of CDEBUG between spin_lock/unlock_irqsave, or spin_lock/unlock_bh could trigger this deadlock. I think a fix could be like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #define cfs_trace_lock_tcd_safe(tcd, flags) ({ int __ret; spin_lock_irqsave(&amp;cfs_trace_glock, flags); __ret = cfs_trace_lock_tcd(tcd); __ret; }) #define cfs_trace_unlock_tcd_safe(tcd, flags) ({ cfs_trace_unlock_tcd(tcd); spin_unlock_irqrestore(&amp;cfs_trace_glock, flags); }) #define cfs_tcd_for_each_type_lock(..., flags) ...... cfs_trace_lock_tcd_safe(tcd, fags); cfs_trace_unlock_tcd(tcd, flags), i++) </pre> </div></div> <p>how do you think?</p> <p>Liang</p> <p>Oleg will construct a patch</p> <p>Patch at <a href="http://review.whamcloud.com/2605" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2605</a></p> <p>I now get this error when testing with the patch.</p> <p>Apr 30 08:19:54 barry-oss4 kernel: <span class="error">&#91;247732.876333&#93;</span> Lustre: debug daemon will attempt to start writing to /chexport/lustre/logs/lustre-log.barry-oss4 (2097152kB m<br/> ax)<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.213070&#93;</span> wanted to write 1008 but wrote -5<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.225887&#93;</span> LNetError: 20124:0<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/>/data/buildsystem/jsimmons-wc/rpmbuild/BUILD/lustre-2.2.0/libcfs/libcfs/tracefile.c:1038:t<br/> racefiled()) ASSERTION(tage-&gt;page != NULL) failed <br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.249433&#93;</span> Kernel panic - not syncing: Lustre debug assertion failure<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.249434&#93;</span> <br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.282160&#93;</span> Pid: 20124, comm: ktracefiled Not tainted 2.6.32-220.el6.wc.x86_64 #1<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.297951&#93;</span> Call Trace:<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.308593&#93;</span> <span class="error">&#91;&lt;ffffffff814ec681&gt;&#93;</span> ? panic+0x78/0x143<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.321844&#93;</span> <span class="error">&#91;&lt;ffffffff8109b56a&gt;&#93;</span> ? do_gettimeofday+0x1a/0x50<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.335818&#93;</span> <span class="error">&#91;&lt;ffffffffa07ff004&gt;&#93;</span> ? cfs_trace_assertion_failed+0x74/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.351479&#93;</span> <span class="error">&#91;&lt;ffffffffa0800edb&gt;&#93;</span> ? tracefiled+0x43b/0x530 <span class="error">&#91;libcfs&#93;</span><br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.365829&#93;</span> <span class="error">&#91;&lt;ffffffff811245b9&gt;&#93;</span> ? free_pages+0x49/0x50<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.379090&#93;</span> <span class="error">&#91;&lt;ffffffff8105fa50&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.393366&#93;</span> <span class="error">&#91;&lt;ffffffffa0800aa0&gt;&#93;</span> ? tracefiled+0x0/0x530 <span class="error">&#91;libcfs&#93;</span><br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.407430&#93;</span> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.420375&#93;</span> <span class="error">&#91;&lt;ffffffffa0800aa0&gt;&#93;</span> ? tracefiled+0x0/0x530 <span class="error">&#91;libcfs&#93;</span><br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.434357&#93;</span> <span class="error">&#91;&lt;ffffffffa0800aa0&gt;&#93;</span> ? tracefiled+0x0/0x530 <span class="error">&#91;libcfs&#93;</span><br/> Apr 30 08:21:41 barry-oss4 kernel: <span class="error">&#91;247839.448139&#93;</span> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>The load on all OSS goes to 300+.</p> <p>Landed for 2.3</p> Development Rank 1|hzv6rr: Rank (Obsolete) 4631 Severity