https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3596 Lustre <p>We had a few OSSes crash at NOAA recently due to an NMI deadlock detected error. I was able to get a vmcore from one and analyze it, and it looks like we are hitting <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>:<br/> crash&gt; bt<br/> PID: 15515 TASK: ffff810c39c237e0 CPU: 0 COMMAND: "kiblnd_connd"<br/> #0 <span class="error">&#91;ffffffff804b8dc0&#93;</span> crash_kexec at ffffffff800b1192<br/> #1 <span class="error">&#91;ffffffff804b8e80&#93;</span> die_nmi at ffffffff80065285<br/> #2 <span class="error">&#91;ffffffff804b8ea0&#93;</span> nmi_watchdog_tick at ffffffff80065a66<br/> #3 <span class="error">&#91;ffffffff804b8ef0&#93;</span> default_do_nmi at ffffffff80065609<br/> #4 <span class="error">&#91;ffffffff804b8f40&#93;</span> do_nmi at ffffffff800658f1<br/> #5 <span class="error">&#91;ffffffff804b8f50&#93;</span> nmi at ffffffff80064ecf<br/> <span class="error">&#91;exception RIP: __write_lock_failed+15&#93;</span><br/> RIP: ffffffff80062197 RSP: ffff81061978dc90 RFLAGS: 00000087<br/> RAX: ffffc20000000000 RBX: ffff8107eb940140 RCX: 0000000000000001<br/> RDX: 0000000000006000 RSI: 0000000000000003 RDI: ffffffff8032d42c<br/> RBP: ffffc20000000000 R8: 0000000000000000 R9: ffff810c39c237e0<br/> R10: ffff8105645d2000 R11: 0000000000002000 R12: ffffffffffffffff<br/> R13: 0000000000007000 R14: 0000000000000001 R15: 0000000000000002<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff81061978dc90&#93;</span> __write_lock_failed at ffffffff80062197<br/> #7 <span class="error">&#91;ffff81061978dc90&#93;</span> _write_lock at ffffffff80064a7d<br/> #8 <span class="error">&#91;ffff81061978dc98&#93;</span> __get_vm_area_node at ffffffff800d51c1<br/> #9 <span class="error">&#91;ffff81061978dcd8&#93;</span> __vmalloc_node at ffffffff800d5952<br/> #10 <span class="error">&#91;ffff81061978dcf8&#93;</span> kiblnd_create_tx_pool at ffffffff8b127e0e<br/> #11 <span class="error">&#91;ffff81061978dd68&#93;</span> kiblnd_pool_alloc_node at ffffffff8b1247a9<br/> #12 <span class="error">&#91;ffff81061978ddc8&#93;</span> kiblnd_get_idle_tx at ffffffff8b12d8d0<br/> #13 <span class="error">&#91;ffff81061978ddd8&#93;</span> kiblnd_check_sends at ffffffff8b12e9b7<br/> #14 <span class="error">&#91;ffff81061978dde8&#93;</span> kiblnd_check_txs at ffffffff8b12c22c<br/> #15 <span class="error">&#91;ffff81061978de48&#93;</span> kiblnd_check_conns at ffffffff8b12ea68<br/> #16 <span class="error">&#91;ffff81061978dea8&#93;</span> kiblnd_connd at ffffffff8b136063<br/> #17 <span class="error">&#91;ffff81061978df48&#93;</span> kernel_thread at ffffffff8005dfc1</p> <p>It appears that <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> was identified for 1.8.x, but never landed on it. If this crash is related to that bug, would it be possible to get an updated patch for potential inclusion on the next 1.8?</p> <p>Thanks.</p> LU-3596

deadlock in kiblnd

Bug Minor Resolved Fixed Bruno Faccini Oz Rentas Tue, 16 Jul 2013 15:32:39 +0000 Fri, 1 Sep 2017 14:57:59 +0000 Fri, 1 Sep 2017 14:57:59 +0000 Lustre 1.8.9 0 6 <p>Hello Kit,<br/> I take ownership of this ticket, since it looks like a new occurence/dup of <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> where I have been somewhat involved with !!...<br/> Since you have a crash-dump for this dead-lock, and just to double-check if it is still the same problem, can you check if there is a thread beeing runnable but not scheduled since a long time and get its stack-trace for me ?</p> <p>Hi Bruno,</p> <p>All the last_rans from the runnables seem similar:<br/> last_ran = 2233784280351367, <br/> last_ran = 2233784307286260, <br/> last_ran = 2233784366546510, <br/> last_ran = 2233784494221153, <br/> last_ran = 2233784499282262, <br/> last_ran = 2233784500452337, <br/> last_ran = 2233784508413095, <br/> last_ran = 2233784520565286, <br/> last_ran = 2233784548748409, <br/> last_ran = 2233784548788509, <br/> last_ran = 2233784588049211, <br/> last_ran = 2233795686537779, <br/> last_ran = 2233796503806272, <br/> last_ran = 2233796699153227, <br/> last_ran = 2233805294134249, <br/> last_ran = 2233805294247424, <br/> last_ran = 2233805294287834, </p> <p>Is that the correct field in task to be looking at?</p> <p>This is the bt for the oldest last_ran:<br/> crash&gt; bt 28352<br/> PID: 28352 TASK: ffff810bd8f01860 CPU: 0 COMMAND: "ll_ost_137"<br/> #0 <span class="error">&#91;ffff810bb7ebbd50&#93;</span> schedule at ffffffff80062fa0<br/> #1 <span class="error">&#91;ffff810bb7ebbe28&#93;</span> ptlrpc_wait_event at ffffffff8b0afee5<br/> #2 <span class="error">&#91;ffff810bb7ebbeb8&#93;</span> ptlrpc_main at ffffffff8b0b03eb<br/> #3 <span class="error">&#91;ffff810bb7ebbf48&#93;</span> kernel_thread at ffffffff8005dfc1</p> <p>I looked through the runnable kiblnd_sd_* tasks (there were only 2), and saw this, which looks like <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>:<br/> crash&gt; bt 15504<br/> PID: 15504 TASK: ffff810c390da0c0 CPU: 17 COMMAND: "kiblnd_sd_13"<br/> #0 <span class="error">&#91;ffff810c3fd3af20&#93;</span> crash_nmi_callback at ffffffff8007c396<br/> #1 <span class="error">&#91;ffff810c3fd3af40&#93;</span> do_nmi at ffffffff800658e5<br/> #2 <span class="error">&#91;ffff810c3fd3af50&#93;</span> nmi at ffffffff80064ecf<br/> <span class="error">&#91;exception RIP: __write_lock_failed+9&#93;</span><br/> RIP: ffffffff80062191 RSP: ffff810619ad7db0 RFLAGS: 00000087<br/> RAX: 0000000000000286 RBX: ffff8105db4da0c0 RCX: 0000000000000100<br/> RDX: 000000000000029a RSI: 00000000fffffffb RDI: ffffffff8b14d39c<br/> RBP: ffff8105db4da0c0 R8: 0000000000000001 R9: 00000000ffffffff<br/> R10: 0000000000000000 R11: 000000000000000a R12: 00000000fffffffb<br/> R13: 0000000000000001 R14: ffffc20010d5fd10 R15: 000000000000000c<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #3 <span class="error">&#91;ffff810619ad7db0&#93;</span> __write_lock_failed at ffffffff80062191<br/> #4 <span class="error">&#91;ffff810619ad7db0&#93;</span> _write_lock_irqsave at ffffffff80064b83<br/> #5 <span class="error">&#91;ffff810619ad7db8&#93;</span> kiblnd_close_conn at ffffffff8b12c844<br/> #6 <span class="error">&#91;ffff810619ad7dd8&#93;</span> kiblnd_tx_complete at ffffffff8b12f20d<br/> #7 <span class="error">&#91;ffff810619ad7e58&#93;</span> kiblnd_scheduler at ffffffff8b136769<br/> #8 <span class="error">&#91;ffff810619ad7f48&#93;</span> kernel_thread at ffffffff8005dfc1</p> <p>Here is the only other runnable IB related thread:<br/> PID: 9331 TASK: ffff81063f2707a0 CPU: 12 COMMAND: "ipoib"<br/> #0 <span class="error">&#91;ffff810c3fc43f20&#93;</span> crash_nmi_callback at ffffffff8007c396<br/> #1 <span class="error">&#91;ffff810c3fc43f40&#93;</span> do_nmi at ffffffff800658e5<br/> #2 <span class="error">&#91;ffff810c3fc43f50&#93;</span> nmi at ffffffff80064ecf<br/> <span class="error">&#91;exception RIP: __smp_call_function_many+150&#93;</span><br/> RIP: ffffffff80077925 RSP: ffff810c3dde7c30 RFLAGS: 00000297<br/> RAX: 0000000000000016 RBX: ffff810c3dde7cb0 RCX: 0000000000000000<br/> RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0<br/> RBP: 0000000000000017 R8: 0000000000000018 R9: 0000000000000028<br/> R10: ffff810c3dde7bd0 R11: ffff8106293e7600 R12: 0000000000000001<br/> R13: 0000000000000000 R14: ffffffff800777f9 R15: ffff810c3dde7cb0<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #3 <span class="error">&#91;ffff810c3dde7c30&#93;</span> __smp_call_function_many at ffffffff80077925<br/> #4 <span class="error">&#91;ffff810c3dde7c78&#93;</span> smp_call_function_many at ffffffff80077a27<br/> #5 <span class="error">&#91;ffff810c3dde7ca8&#93;</span> smp_call_function at ffffffff80077b18<br/> #6 <span class="error">&#91;ffff810c3dde7cf8&#93;</span> on_each_cpu at ffffffff80096bcd<br/> #7 <span class="error">&#91;ffff810c3dde7d18&#93;</span> __remove_vm_area at ffffffff800d55f8<br/> #8 <span class="error">&#91;ffff810c3dde7d38&#93;</span> remove_vm_area at ffffffff800d5627<br/> #9 <span class="error">&#91;ffff810c3dde7d48&#93;</span> __vunmap at ffffffff800d5673<br/> #10 <span class="error">&#91;ffff810c3dde7d68&#93;</span> mlx4_buf_free at ffffffff884f8352<br/> #11 <span class="error">&#91;ffff810c3dde7d88&#93;</span> mlx4_ib_destroy_qp at ffffffff88542219<br/> #12 <span class="error">&#91;ffff810c3dde7dc8&#93;</span> ib_destroy_qp at ffffffff884a1bb0<br/> #13 <span class="error">&#91;ffff810c3dde7df8&#93;</span> ipoib_cm_tx_reap at ffffffff88741999<br/> #14 <span class="error">&#91;ffff810c3dde7e38&#93;</span> run_workqueue at ffffffff8004d8ac<br/> #15 <span class="error">&#91;ffff810c3dde7e78&#93;</span> worker_thread at ffffffff8004a1e5<br/> #16 <span class="error">&#91;ffff810c3dde7ee8&#93;</span> kthread at ffffffff80032c45<br/> #17 <span class="error">&#91;ffff810c3dde7f48&#93;</span> kernel_thread at ffffffff8005dfc1</p> <p>Thanks.</p> <p>Hello Cory,<br/> I see only 17 runnable (running?) threads, how many core is your OSS ? I presume 16, and all are in NMI context with the 17th being pid 28352?<br/> Can you check if you get the same order/threads with "ps -l" output in crash ? <br/> Also can you provide the "dmesg/log"<ins>"ps -l"</ins>"foreach bt" outputs from crash too ? Or may be the crash-dump itself if site policy is ok ?</p> <p>Hi Bruno, </p> <p>There are actually 24 CPUs. I may have misinterpreted running vs runnable. I looked for all the tasks with RU in ps, but is that running? I've attached the requested output. </p> <p>The vmcore is 2GB, is there a server I could put it on? </p> <p>Thanks,<br/> Kit</p> <p>RU in ps stands for RUnnable and not only running. That was my wish to get all runnable vs only the running ones. <br/> Humm, but if there are 24 cores you should see at least 24 RUs including some swapper/idle threads. Thus, I definitely need the 3 crash sub-commands outputs I earlier requested, you said you attached them but I don't see them ? <br/> Concerning the crash-dump upload I have sent you the instructions privately.</p> <p>Having a look to crash-dump extracts you posted in lu-3596.out file, I am less and less convinced it is a dup of <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>. This is because in <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> dead-lock occurred due to lock owner re-schedule and lock competitor elected on CPU, but here lock owner is stuck waiting for a TLB-flush IPI broadcast completion ...</p> <p>Also, is the OSS running wit /proc/sys/kernel/nmi_watchdog set ?</p> <p>Ok, from the crash-dump, scenario is different than for <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>, there are 3 threads involved and all 3 spinning on their CPUs with the following stack-traces at the time of the NMI/watchdog :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 9331 TASK: ffff81063f2707a0 CPU: 12 COMMAND: "ipoib" #0 [ffff810c3fc43f20] crash_nmi_callback at ffffffff8007c396 #1 [ffff810c3fc43f40] do_nmi at ffffffff800658e5 #2 [ffff810c3fc43f50] nmi at ffffffff80064ecf [exception RIP: __smp_call_function_many+150] RIP: ffffffff80077925 RSP: ffff810c3dde7c30 RFLAGS: 00000297 RAX: 0000000000000016 RBX: ffff810c3dde7cb0 RCX: 0000000000000000 RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0 RBP: 0000000000000017 R8: 0000000000000018 R9: 0000000000000028 R10: ffff810c3dde7bd0 R11: ffff8106293e7600 R12: 0000000000000001 R13: 0000000000000000 R14: ffffffff800777f9 R15: ffff810c3dde7cb0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #3 [ffff810c3dde7c30] __smp_call_function_many at ffffffff80077925 #4 [ffff810c3dde7c78] smp_call_function_many at ffffffff80077a27 #5 [ffff810c3dde7ca8] smp_call_function at ffffffff80077b18 #6 [ffff810c3dde7cf8] on_each_cpu at ffffffff80096bcd #7 [ffff810c3dde7d18] __remove_vm_area at ffffffff800d55f8 #8 [ffff810c3dde7d38] remove_vm_area at ffffffff800d5627 #9 [ffff810c3dde7d48] __vunmap at ffffffff800d5673 #10 [ffff810c3dde7d68] mlx4_buf_free at ffffffff884f8352 [mlx4_core] #11 [ffff810c3dde7d88] mlx4_ib_destroy_qp at ffffffff88542219 [mlx4_ib] #12 [ffff810c3dde7dc8] ib_destroy_qp at ffffffff884a1bb0 [ib_core] #13 [ffff810c3dde7df8] ipoib_cm_tx_reap at ffffffff88741999 [ib_ipoib] #14 [ffff810c3dde7e38] run_workqueue at ffffffff8004d8ac #15 [ffff810c3dde7e78] worker_thread at ffffffff8004a1e5 #16 [ffff810c3dde7ee8] kthread at ffffffff80032c45 #17 [ffff810c3dde7f48] kernel_thread at ffffffff8005dfc1 PID: 15504 TASK: ffff810c390da0c0 CPU: 17 COMMAND: "kiblnd_sd_13" #0 [ffff810c3fd3af20] crash_nmi_callback at ffffffff8007c396 #1 [ffff810c3fd3af40] do_nmi at ffffffff800658e5 #2 [ffff810c3fd3af50] nmi at ffffffff80064ecf [exception RIP: __write_lock_failed+9] RIP: ffffffff80062191 RSP: ffff810619ad7db0 RFLAGS: 00000087 RAX: 0000000000000286 RBX: ffff8105db4da0c0 RCX: 0000000000000100 RDX: 000000000000029a RSI: 00000000fffffffb RDI: ffffffff8b14d39c RBP: ffff8105db4da0c0 R8: 0000000000000001 R9: 00000000ffffffff R10: 0000000000000000 R11: 000000000000000a R12: 00000000fffffffb R13: 0000000000000001 R14: ffffc20010d5fd10 R15: 000000000000000c ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #3 [ffff810619ad7db0] __write_lock_failed at ffffffff80062191 #4 [ffff810619ad7db0] _write_lock_irqsave at ffffffff80064b83 #5 [ffff810619ad7db8] kiblnd_close_conn at ffffffff8b12c844 [ko2iblnd] #6 [ffff810619ad7dd8] kiblnd_tx_complete at ffffffff8b12f20d [ko2iblnd] #7 [ffff810619ad7e58] kiblnd_scheduler at ffffffff8b136769 [ko2iblnd] #8 [ffff810619ad7f48] kernel_thread at ffffffff8005dfc1 PID: 15515 TASK: ffff810c39c237e0 CPU: 0 COMMAND: "kiblnd_connd" #0 [ffffffff804b8dc0] crash_kexec at ffffffff800b1192 #1 [ffffffff804b8e80] die_nmi at ffffffff80065285 #2 [ffffffff804b8ea0] nmi_watchdog_tick at ffffffff80065a66 #3 [ffffffff804b8ef0] default_do_nmi at ffffffff80065609 #4 [ffffffff804b8f40] do_nmi at ffffffff800658f1 #5 [ffffffff804b8f50] nmi at ffffffff80064ecf [exception RIP: __write_lock_failed+15] RIP: ffffffff80062197 RSP: ffff81061978dc90 RFLAGS: 00000087 RAX: ffffc20000000000 RBX: ffff8107eb940140 RCX: 0000000000000001 RDX: 0000000000006000 RSI: 0000000000000003 RDI: ffffffff8032d42c RBP: ffffc20000000000 R8: 0000000000000000 R9: ffff810c39c237e0 R10: ffff8105645d2000 R11: 0000000000002000 R12: ffffffffffffffff R13: 0000000000007000 R14: 0000000000000001 R15: 0000000000000002 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #6 [ffff81061978dc90] __write_lock_failed at ffffffff80062197 #7 [ffff81061978dc90] _write_lock at ffffffff80064a7d #8 [ffff81061978dc98] __get_vm_area_node at ffffffff800d51c1 #9 [ffff81061978dcd8] __vmalloc_node at ffffffff800d5952 #10 [ffff81061978dcf8] kiblnd_create_tx_pool at ffffffff8b127e0e [ko2iblnd] #11 [ffff81061978dd68] kiblnd_pool_alloc_node at ffffffff8b1247a9 [ko2iblnd] #12 [ffff81061978ddc8] kiblnd_get_idle_tx at ffffffff8b12d8d0 [ko2iblnd] #13 [ffff81061978ddd8] kiblnd_check_sends at ffffffff8b12e9b7 [ko2iblnd] #14 [ffff81061978dde8] kiblnd_check_txs at ffffffff8b12c22c [ko2iblnd] #15 [ffff81061978de48] kiblnd_check_conns at ffffffff8b12ea68 [ko2iblnd] #16 [ffff81061978dea8] kiblnd_connd at ffffffff8b136063 [ko2iblnd] #17 [ffff81061978df48] kernel_thread at ffffffff8005dfc1 </pre> </div></div> <p>where :</p> <p> _ 9331/"ipoib" thread owns vmlist_lock, during mlx4 driver buffer unmap, but then wants all others CPUs to flush their TLB and thus sent an IPI to each of them and it is now busy waiting for one reluctant peer ...</p> <p> _ 15504/"kiblnd_sd_13" thread is spinning/waiting on kiblnd_data.kib_global_lock in kiblnd_close_conn() but with interrupts disabled, and thus is likely to be the one still missing the IPI and to flush its TLB !!</p> <p> _ 15515/"kiblnd_connd" thread owns kiblnd_data.kib_global_lock since kiblnd_check_conns() but is now spinning/waiting for vmlist_lock due to kmem alloc in kiblnd_create_tx_pool().</p> <p>So 9331 blocks 15515 who blocks 15504 who blocks 9331 !!</p> <p>But even if it is a new scenario than the 2 already described in <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>, I can now agree that it would also be fixed by the same patch since with it thread 15515 will not spin on vmlist_lock when already owning kiblnd_data.kib_global_lock !!</p> <p>Thanks for the analysis! I have ported the patch from <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> to b1_8:<br/> <a href="http://review.whamcloud.com/7092" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7092</a></p> <p>Does that look correct?</p> <p>Yes, patch/back-port looks ok. Need to wait Liang's feed-back to confirm. Also I had to re-trigger auto-tests due to un-related failure.</p> <p>This is resolved. Please close.</p> <p>Thanks Oz</p> Development End date Fri, 9 May 2014 15:32:39 +0000 Rank 1|hzvvhb: Rank (Obsolete) 9116 Severity Start date Tue, 16 Jul 2013 15:32:39 +0000