https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-78 Lustre <p>kiblnd_check_conns() called kiblnd_check_sends() with read_lock(&amp;kiblnd_data.kib_global_lock), it's<br/> so wrong because kiblnd_check_sends() could do a lot of things:<br/> 1. lnet_finalize(), which is not safe with spinlock<br/> 2. kiblnd_close_conn(), which will hold write_lock(&amp;kiblnd_data.kib_global_lock) to be bomb of<br/> deadlock</p> <p>There is a patch in the bug from Isaac, but it introduced som regressions and was therefore reverted.</p> LU-78

kiblnd_check_conns can deadlock

Bug Major Resolved Fixed Liang Zhen Oleg Drokin paj Wed, 9 Feb 2011 13:00:21 +0000 Sun, 8 Apr 2012 15:12:27 +0000 Mon, 5 Mar 2012 20:36:47 +0000 Lustre 2.0.0 Lustre 1.8.6 Lustre 2.2.0 Lustre 2.1.2 0 8 <p>Hello,</p> <p>At TGCC, running Lustre 2.1.0, we have encountered multiple occurences where an OSS was so loaded that a lot of Client were complaining against it and we were unable to logon to it.</p> <p>A crash-dump was forced and it analysis alway show the same situation were almost all CPUs/cores are stuck running kiblnd_sd_&lt;id&gt; and/or ib_cm/&lt;id&gt; threads all spinning around the "kiblnd_data.kib_global_lock" RW-SpinLock, trying to write-locking it, with Kernel stacks/trace-backs like following :<br/> =====================================================================<br/> PID: 10493 TASK: ffff88087a4ae790 CPU: 6 COMMAND: "kiblnd_sd_23"<br/> #0 <span class="error">&#91;ffff88088e447e90&#93;</span> crash_nmi_callback at ffffffff8101fd06<br/> #1 <span class="error">&#91;ffff88088e447ea0&#93;</span> notifier_call_chain at ffffffff814837f5<br/> #2 <span class="error">&#91;ffff88088e447ee0&#93;</span> atomic_notifier_call_chain at ffffffff8148385a<br/> #3 <span class="error">&#91;ffff88088e447ef0&#93;</span> notify_die at ffffffff8108026e<br/> #4 <span class="error">&#91;ffff88088e447f20&#93;</span> do_nmi at ffffffff81481443<br/> #5 <span class="error">&#91;ffff88088e447f50&#93;</span> nmi at ffffffff81480d50<br/> <span class="error">&#91;exception RIP: __write_lock_failed+9&#93;</span><br/> RIP: ffffffff81264919 RSP: ffff880874453d68 RFLAGS: 00000087<br/> RAX: 0000000000000246 RBX: ffff880f5a797ad8 RCX: 0000000000000078<br/> RDX: 0000000000000246 RSI: 00000000000000d1 RDI: ffffffffa0572ecc<br/> RBP: ffff880874453d70 R8: ffff880d9a408000 R9: 0000000000000012<br/> R10: dead000000100100 R11: 0000000000000001 R12: ffff880ec7518b80<br/> R13: ffff88087cd4a8c0 R14: 0000000000000000 R15: ffff88087d31ee00<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff880874453d68&#93;</span> __write_lock_failed at ffffffff81264919<br/> #7 <span class="error">&#91;ffff880874453d68&#93;</span> _write_lock_irqsave at ffffffff814805e8<br/> #8 <span class="error">&#91;ffff880874453d78&#93;</span> kiblnd_rx_complete at ffffffffa056682d <span class="error">&#91;ko2iblnd&#93;</span><br/> #9 <span class="error">&#91;ffff880874453df8&#93;</span> kiblnd_complete at ffffffffa05669f2 <span class="error">&#91;ko2iblnd&#93;</span><br/> #10 <span class="error">&#91;ffff880874453e38&#93;</span> kiblnd_scheduler at ffffffffa0566d94 <span class="error">&#91;ko2iblnd&#93;</span><br/> #11 <span class="error">&#91;ffff880874453f48&#93;</span> kernel_thread at ffffffff810041aa</p> <p>PID: 9312 TASK: ffff88087cd5c080 CPU: 25 COMMAND: "ib_cm/25"<br/> #0 <span class="error">&#91;ffff88048e587e90&#93;</span> crash_nmi_callback at ffffffff8101fd06<br/> #1 <span class="error">&#91;ffff88048e587ea0&#93;</span> notifier_call_chain at ffffffff814837f5<br/> #2 <span class="error">&#91;ffff88048e587ee0&#93;</span> atomic_notifier_call_chain at ffffffff8148385a<br/> #3 <span class="error">&#91;ffff88048e587ef0&#93;</span> notify_die at ffffffff8108026e<br/> #4 <span class="error">&#91;ffff88048e587f20&#93;</span> do_nmi at ffffffff81481443<br/> #5 <span class="error">&#91;ffff88048e587f50&#93;</span> nmi at ffffffff81480d50<br/> <span class="error">&#91;exception RIP: __write_lock_failed+9&#93;</span><br/> RIP: ffffffff81264919 RSP: ffff88087a033c58 RFLAGS: 00000087<br/> RAX: 0000000000000282 RBX: ffff8810681b41c0 RCX: 0000000000000002<br/> RDX: 0000000000000282 RSI: 0000000000000000 RDI: ffffffffa0572ecc<br/> RBP: ffff88087a033c60 R8: 0000000000000000 R9: 0000000000000000<br/> R10: 0000000000000000 R11: 0000000000000001 R12: ffff880f2291a000<br/> R13: 0000000000000000 R14: ffff8807a4210200 R15: ffffe8f7ff997988<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff88087a033c58&#93;</span> __write_lock_failed at ffffffff81264919<br/> #7 <span class="error">&#91;ffff88087a033c58&#93;</span> _write_lock_irqsave at ffffffff814805e8<br/> #8 <span class="error">&#91;ffff88087a033c68&#93;</span> kiblnd_close_conn at ffffffffa055fa4b <span class="error">&#91;ko2iblnd&#93;</span><br/> #9 <span class="error">&#91;ffff88087a033c98&#93;</span> kiblnd_cm_callback at ffffffffa056a910 <span class="error">&#91;ko2iblnd&#93;</span><br/> #10 <span class="error">&#91;ffff88087a033d08&#93;</span> cma_ib_handler at ffffffffa03686e9 <span class="error">&#91;rdma_cm&#93;</span><br/> #11 <span class="error">&#91;ffff88087a033d88&#93;</span> cm_process_work at ffffffffa0346e67 <span class="error">&#91;ib_cm&#93;</span><br/> #12 <span class="error">&#91;ffff88087a033dd8&#93;</span> cm_work_handler at ffffffffa03493c9 <span class="error">&#91;ib_cm&#93;</span><br/> #13 <span class="error">&#91;ffff88087a033e38&#93;</span> worker_thread at ffffffff810749a0<br/> #14 <span class="error">&#91;ffff88087a033ee8&#93;</span> kthread at ffffffff81079f36<br/> #15 <span class="error">&#91;ffff88087a033f48&#93;</span> kernel_thread at ffffffff810041aa<br/> =====================================================================</p> <p>and this because the "kiblnd_data.kib_global_lock" read-locker is awaiting to be scheduled on its CPU/Core sinc a very long time with the following Kernel-stack :<br/> =====================================================================<br/> PID: 10494 TASK: ffff88087a4ae040 CPU: 19 COMMAND: "kiblnd_connd"<br/> #0 <span class="error">&#91;ffff88083b577c10&#93;</span> schedule at ffffffff8147dddc<br/> #1 <span class="error">&#91;ffff88083b577cd8&#93;</span> cfs_schedule at ffffffffa03fd74e <span class="error">&#91;libcfs&#93;</span><br/> #2 <span class="error">&#91;ffff88083b577ce8&#93;</span> kiblnd_pool_alloc_node at ffffffffa05569b3 <span class="error">&#91;ko2iblnd&#93;</span><br/> #3 <span class="error">&#91;ffff88083b577d58&#93;</span> kiblnd_get_idle_tx at ffffffffa056063d <span class="error">&#91;ko2iblnd&#93;</span><br/> #4 <span class="error">&#91;ffff88083b577d88&#93;</span> kiblnd_check_sends at ffffffffa0562ed6 <span class="error">&#91;ko2iblnd&#93;</span><br/> #5 <span class="error">&#91;ffff88083b577e18&#93;</span> kiblnd_check_conns at ffffffffa0563123 <span class="error">&#91;ko2iblnd&#93;</span><br/> #6 <span class="error">&#91;ffff88083b577e98&#93;</span> kiblnd_connd at ffffffffa05664f7 <span class="error">&#91;ko2iblnd&#93;</span><br/> #7 <span class="error">&#91;ffff88083b577f48&#93;</span> kernel_thread at ffffffff810041aa<br/> =====================================================================</p> <p>and this because the CPU/Core it has been scheduled to is starved by one of its write-lock competitor.</p> <p>This scenario definitelly seems to be a specific case/consequence of the bug/limitation described in this <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>, but since it now happpens in the "real life" on a customer site running production workload we would like to have its priority to be raised.</p> <p>Do you agree with both my analysis and request ???</p> <p>Possible ways to fix this problem could be to :</p> <p> _ avoid re-schedule()/yield CPU in kiblnd_pool_alloc_node(), but wait/spin active.</p> <p> _ change kiblnd_data.kib_global_lock RW-SpinLock to a Semaphore.</p> <p>What do you think ??</p> <p>I've posted a patch at here:<br/> <a href="http://review.whamcloud.com/#change,2166" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2166</a></p> <p>Since it will allow kiblnd_pool_alloc_node() to re-schedule() without keeping a read-lock on kiblnd_data.kib_global_lock, this patch looks good to me.</p> <p>On the other hand, this problem is very solid/frequent at TGCC, so do you think this patch will be exposed/tested soon ???</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=ofa/493/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,ofa #493</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision cc875104bb81313415167425ce21c562ddf540c9)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=cc875104bb81313415167425ce21c562ddf540c9" class="external-link" target="_blank" rel="nofollow noopener">cc875104bb81313415167425ce21c562ddf540c9</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Our production systems got hit by this problem also.</p> <p>Jay, the patch is already in master now and I'm going to close this ticket.</p> <p>patch landed and close it</p> <p>Ah, I missed your "review" link and though it had not gone through maloo yet.<br/> Cool, I will pick it up! Thanks!</p> <p>Cherry-pick into b2_1 was smooth, but failed in b1_8 tree.<br/> Do you plan to backport the change to b1_8? We got hit with this problem with our 1.8.6 servers twice last week.</p> <p>Not sure it is the ideal way to do so, but I was not sure it need a new JIRA creation since this update is mainly for information ...</p> <p>So just for info, during a multi-OSSes hang situation mainly reproducing <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a>'s problem, I found an other, quite similar but different, dead-lock scenario whith the following details :</p> <p> _ all CPUs are busy running either ib_cm/&lt;id&gt; or kiblnd_sd_&lt;id&gt; and the kiblnd_connd which, this time, is not hung waiting to be re-scheduled on a starved CPU but instead live-lock spinning on a (struct kib_poolset *)<del>&gt;ps_lock, like several ib_cm/&lt;id&gt; or kiblnd_sd_&lt;id&gt; other threads, due to (struct kib_poolset *)</del>&gt;ps_increasing set, and/but still owning kiblnd_data.kib_global_lock from kiblnd_check_conns() :<br/> ==========================================================================<br/> PID: 15371 TASK: ffff881067135100 CPU: 26 COMMAND: "kiblnd_connd"<br/> #0 <span class="error">&#91;ffff88088e587e90&#93;</span> crash_nmi_callback at ffffffff8101fd06<br/> 0000001 <span class="error">&#91;ffff88088e587ea0&#93;</span> notifier_call_chain at ffffffff814837f5<br/> 0000002 <span class="error">&#91;ffff88088e587ee0&#93;</span> atomic_notifier_call_chain at ffffffff8148385a<br/> 0000003 <span class="error">&#91;ffff88088e587ef0&#93;</span> notify_die at ffffffff8108026e<br/> 0000004 <span class="error">&#91;ffff88088e587f20&#93;</span> do_nmi at ffffffff81481443<br/> 0000005 <span class="error">&#91;ffff88088e587f50&#93;</span> nmi at ffffffff81480d50<br/> <span class="error">&#91;exception RIP: _spin_lock+30&#93;</span><br/> RIP: ffffffff8148062e RSP: ffff88107cfb3ce0 RFLAGS: 00000287<br/> RAX: 0000000000000987 RBX: ffff88107c04ec30 RCX: 0000000000000000<br/> RDX: 000000000000097c RSI: ffff88088e592e10 RDI: ffff88107c04ec30<br/> RBP: ffff88107cfb3ce0 R8: 0000000000000001 R9: 00000000ffffffff<br/> R10: 0000000000000000 R11: 0000000000000001 R12: ffff88107c04ec60<br/> R13: ffff88107c04ec40 R14: ffff88107cfb3d18 R15: 0000000000000001<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> 0000006 <span class="error">&#91;ffff88107cfb3ce0&#93;</span> _spin_lock at ffffffff8148062e<br/> 0000007 <span class="error">&#91;ffff88107cfb3ce8&#93;</span> kiblnd_pool_alloc_node at ffffffffa055c7c8 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000008 <span class="error">&#91;ffff88107cfb3d58&#93;</span> kiblnd_get_idle_tx at ffffffffa056663d <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000009 <span class="error">&#91;ffff88107cfb3d88&#93;</span> kiblnd_check_sends at ffffffffa0568ed6 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000010 <span class="error">&#91;ffff88107cfb3e18&#93;</span> kiblnd_check_conns at ffffffffa0569123 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000011 <span class="error">&#91;ffff88107cfb3e98&#93;</span> kiblnd_connd at ffffffffa056c4f7 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000012 <span class="error">&#91;ffff88107cfb3f48&#93;</span> kernel_thread at ffffffff810041aa<br/> ==========================================================================</p> <p> _ the thread who did set "ps_increasing" in kiblnd_pool_alloc_node() has been re-scheduled during kmalloc(), but is unable to get-back on its CPU/Core due to "ib_cm/29" thread now "write-lock" spinning on kiblnd_data.kib_global_lock, both with the following stacks :<br/> ========================================================================<br/> PID: 15354 TASK: ffff881067145850 CPU: 29 COMMAND: "kiblnd_sd_07"<br/> #0 <span class="error">&#91;ffff88107d0b7590&#93;</span> schedule at ffffffff8147dddc<br/> 0000001 <span class="error">&#91;ffff88107d0b7658&#93;</span> __cond_resched at ffffffff8104d44a<br/> 0000002 <span class="error">&#91;ffff88107d0b7678&#93;</span> _cond_resched at ffffffff8147e680<br/> 0000003 <span class="error">&#91;ffff88107d0b7688&#93;</span> kmem_cache_alloc_node_notrace at ffffffff811466d8<br/> 0000004 <span class="error">&#91;ffff88107d0b76c8&#93;</span> __kmalloc_node at ffffffff811468eb<br/> 0000005 <span class="error">&#91;ffff88107d0b7718&#93;</span> __vmalloc_area_node at ffffffff81133c2f<br/> 0000006 <span class="error">&#91;ffff88107d0b7778&#93;</span> __vmalloc_node at ffffffff81133bc2<br/> 0000007 <span class="error">&#91;ffff88107d0b77b8&#93;</span> vmalloc at ffffffff81133e7c<br/> 0000008 <span class="error">&#91;ffff88107d0b77c8&#93;</span> cfs_alloc_large at ffffffffa040399e <span class="error">&#91;libcfs&#93;</span><br/> 0000009 <span class="error">&#91;ffff88107d0b77d8&#93;</span> kiblnd_create_tx_pool at ffffffffa055e649 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000010 <span class="error">&#91;ffff88107d0b7888&#93;</span> kiblnd_pool_alloc_node at ffffffffa055c90f <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000011 <span class="error">&#91;ffff88107d0b78f8&#93;</span> kiblnd_get_idle_tx at ffffffffa056663d <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000012 <span class="error">&#91;ffff88107d0b7928&#93;</span> kiblnd_check_sends at ffffffffa0568ed6 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000013 <span class="error">&#91;ffff88107d0b79b8&#93;</span> kiblnd_post_rx at ffffffffa056b170 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000014 <span class="error">&#91;ffff88107d0b7a48&#93;</span> kiblnd_recv at ffffffffa056b46a <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000015 <span class="error">&#91;ffff88107d0b7b08&#93;</span> lnet_ni_recv at ffffffffa0464608 <span class="error">&#91;lnet&#93;</span><br/> 0000016 <span class="error">&#91;ffff88107d0b7b98&#93;</span> lnet_recv_put at ffffffffa0464966 <span class="error">&#91;lnet&#93;</span><br/> 0000017 <span class="error">&#91;ffff88107d0b7be8&#93;</span> lnet_parse at ffffffffa046b32f <span class="error">&#91;lnet&#93;</span><br/> 0000018 <span class="error">&#91;ffff88107d0b7ce8&#93;</span> kiblnd_handle_rx at ffffffffa056bb8b <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000019 <span class="error">&#91;ffff88107d0b7d78&#93;</span> kiblnd_rx_complete at ffffffffa056c850 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000020 <span class="error">&#91;ffff88107d0b7df8&#93;</span> kiblnd_complete at ffffffffa056c9f2 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000021 <span class="error">&#91;ffff88107d0b7e38&#93;</span> kiblnd_scheduler at ffffffffa056cd94 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000022 <span class="error">&#91;ffff88107d0b7f48&#93;</span> kernel_thread at ffffffff810041aa</p> <p>PID: 9529 TASK: ffff88087dad7850 CPU: 29 COMMAND: "ib_cm/29"<br/> #0 <span class="error">&#91;ffff88048e5c7e90&#93;</span> crash_nmi_callback at ffffffff8101fd06<br/> 0000001 <span class="error">&#91;ffff88048e5c7ea0&#93;</span> notifier_call_chain at ffffffff814837f5<br/> 0000002 <span class="error">&#91;ffff88048e5c7ee0&#93;</span> atomic_notifier_call_chain at ffffffff8148385a<br/> 0000003 <span class="error">&#91;ffff88048e5c7ef0&#93;</span> notify_die at ffffffff8108026e<br/> 0000004 <span class="error">&#91;ffff88048e5c7f20&#93;</span> do_nmi at ffffffff81481443<br/> 0000005 <span class="error">&#91;ffff88048e5c7f50&#93;</span> nmi at ffffffff81480d50<br/> <span class="error">&#91;exception RIP: __write_lock_failed+9&#93;</span><br/> RIP: ffffffff81264919 RSP: ffff880879e3ba28 RFLAGS: 00000087<br/> RAX: 0000000000000246 RBX: ffff880804b90840 RCX: 0000000000000000<br/> RDX: 0000000000000246 RSI: 0000000000000050 RDI: ffffffffa0578ecc<br/> RBP: ffff880879e3ba30 R8: 0000000000000246 R9: 0000000000000010<br/> R10: 00000000787c3001 R11: 0000000000000001 R12: ffff88107cc11600<br/> R13: ffff88107d28fd80 R14: 000500030a653047 R15: ffff880879e3bb90<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> 0000006 <span class="error">&#91;ffff880879e3ba28&#93;</span> __write_lock_failed at ffffffff81264919<br/> 0000007 <span class="error">&#91;ffff880879e3ba28&#93;</span> _write_lock_irqsave at ffffffff814805e8<br/> 0000008 <span class="error">&#91;ffff880879e3ba38&#93;</span> kiblnd_create_peer at ffffffffa05597a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000009 <span class="error">&#91;ffff880879e3bab8&#93;</span> kiblnd_passive_connect at ffffffffa056e9df <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000010 <span class="error">&#91;ffff880879e3bbd8&#93;</span> kiblnd_cm_callback at ffffffffa057059d <span class="error">&#91;ko2iblnd&#93;</span><br/> 0000011 <span class="error">&#91;ffff880879e3bc48&#93;</span> cma_req_handler at ffffffffa03741c0 <span class="error">&#91;rdma_cm&#93;</span><br/> 0000012 <span class="error">&#91;ffff880879e3bd08&#93;</span> cm_process_work at ffffffffa0350e67 <span class="error">&#91;ib_cm&#93;</span><br/> 0000013 <span class="error">&#91;ffff880879e3bd58&#93;</span> cm_req_handler at ffffffffa03527e0 <span class="error">&#91;ib_cm&#93;</span><br/> 0000014 <span class="error">&#91;ffff880879e3bdd8&#93;</span> cm_work_handler at ffffffffa03532d5 <span class="error">&#91;ib_cm&#93;</span><br/> 0000015 <span class="error">&#91;ffff880879e3be38&#93;</span> worker_thread at ffffffff810749a0<br/> 0000016 <span class="error">&#91;ffff880879e3bee8&#93;</span> kthread at ffffffff81079f36<br/> 0000017 <span class="error">&#91;ffff880879e3bf48&#93;</span> kernel_thread at ffffffff810041aa<br/> ========================================================================</p> <p>So, my assumption is that the fix from <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> already provided for the original situation may also fix/avoid this one, again due to now calling kiblnd_check_sends() from kiblnd_check_conns() without locking kiblnd_data.kib_global_lock anymore ...</p> <p>But anyway, I also wanted to report/document this 2nd scenario, and also the potential "live-lock" situation that has been encountered with the following code path in kiblnd_pool_alloc_node() :<br/> =======================================<br/> cfs_list_t *<br/> kiblnd_pool_alloc_node(kib_poolset_t *ps)<br/> {<br/> cfs_list_t *node;<br/> kib_pool_t *pool;<br/> int rc;</p> <p> again:<br/> cfs_spin_lock(&amp;ps-&gt;ps_lock);<br/> cfs_list_for_each_entry(pool, &amp;ps-&gt;ps_pool_list, po_list) {</p> <p>......</p> <p> /* no available tx pool and ... */<br/> if (ps-&gt;ps_increasing) </p> { /* another thread is allocating a new pool */ cfs_spin_unlock(&amp;ps-&gt;ps_lock); CDEBUG(D_NET, "Another thread is allocating new " "%s pool, waiting for her to complete\n", ps-&gt;ps_name); cfs_schedule(); goto again; } <p>.......</p> <p>=======================================<br/> which shoud be changed with a semaphore/wait-queue mechanism ??... </p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,client,sles11,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,client,el6,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,server,el6,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,client,el5,ofa #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,server,el5,ofa #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,client,el6,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,server,el6,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,client,el5,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,server,el5,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,server,el5,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,server,el5,ofa #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » x86_64,client,el5,ofa #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-b2_1/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/41/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b2_1 » i686,client,el5,inkernel #41</a><br/> <a href="https://jira.whamcloud.com/browse/LU-78" title="kiblnd_check_conns can deadlock" class="issue-link" data-issue-key="LU-78"><del>LU-78</del></a> o2iblnd: kiblnd_check_conns can deadlock (Revision dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b2_1&amp;a=commit&amp;h=dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155" class="external-link" target="_blank" rel="nofollow noopener">dcf1b2e6e2f22935f823cb7610b33b8f9c3ef155</a><br/> Files : </p> <ul> <li>lnet/klnds/o2iblnd/o2iblnd.h</li> <li>lnet/klnds/o2iblnd/o2iblnd_cb.c</li> </ul> Bugzilla ID 20288.0 Development Rank 1|hzv78v: Rank (Obsolete) 4708 Severity