https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-290 Lustre <p>It seems that clients can flood a server with reconnect requests<br/> when this one is returning EBUSY because it is still processing<br/> requests from the old connection.</p> <p>e.g. seen on 1.8.2 with a cluster having ~800 clients:</p> <p>Mar 3 20:26:16 md061i kernel: Lustre: 27033:0:(ldlm_lib.c:835:target_handle_connect())<br/> share3-MDT0000: refuse reconnection from eb9b2b28-5e23-8dc1-7024-7810bf8a74ff@173.25.10.184@o2ib to<br/> 0xffff81039e70f000; still busy with 1 active RPCs<br/> Mar 3 20:26:16 md061i kernel: Lustre: 27033:0:(ldlm_lib.c:835:target_handle_connect()) Skipped<br/> 4527 previous similar messages<br/> Mar 3 20:26:18 md061i kernel: Lustre: 27116:0:(ldlm_lib.c:835:target_handle_connect())<br/> share3-MDT0000: refuse reconnection from eb9b2b28-5e23-8dc1-7024-7810bf8a74ff@173.25.10.184@o2ib to<br/> 0xffff81039e70f000; still busy with 1 active RPCs<br/> Mar 3 20:26:18 md061i kernel: Lustre: 27116:0:(ldlm_lib.c:835:target_handle_connect()) Skipped<br/> 10580 previous similar messages</p> <p>From code review, this looks like a side effect of bug 18674.<br/> Since we now bypass import_select_connection() on EBUSY and EAGAIN,<br/> ptlrpc_connect_interpret-&gt;ptlrpc_maybe_ping_import_soon always triggers<br/> an immediate ping causing clients to reconnect in a busy loop.<br/> ------- Comment #1 From Johann Lombardi 2010-03-31 16:01:54</p> LU-290

Reconnects are not throttled

Bug Minor Closed Won't Fix Lai Siyao Lai Siyao Fri, 6 May 2011 22:03:52 +0000 Tue, 16 Aug 2016 16:32:42 +0000 Tue, 16 Aug 2016 16:32:41 +0000 Lustre 2.0.0 Lustre 2.1.0 Sat, 21 May 2011 00:00:00 +0000 0 1 <p>Lai</p> <p>Just to warn you on this one - Oleg was not sure whether this would even be a problem on master so the first step is to establish whether this is before investing time in trying to port the patch</p> <p>Regards</p> <p>Peter</p> <p>I see, I'll investigate first.</p> <p>The comments in bz22423 and current code shows a patch for 2.x was committed, but caused a conf-sanity.sh failure, and then reverted.</p> <p>I'll do some test and find out the cause of that failure.</p> <p>Autotest result looks normal, the patch will be put to review.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/import.c</li> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/import.c</li> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/241/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #241</a><br/> <a href="https://jira.whamcloud.com/browse/LU-290" title="Reconnects are not throttled" class="issue-link" data-issue-key="LU-290"><del>LU-290</del></a> Reconnects are not throttled</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=86b2211e55dcc509da85b21ece8830e2a9b70db1" class="external-link" target="_blank" rel="nofollow noopener">86b2211e55dcc509da85b21ece8830e2a9b70db1</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>lustre/ptlrpc/import.c</li> </ul> <p>please see this:</p> <p><a href="https://maloo.whamcloud.com/test_sets/e098dda6-c262-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e098dda6-c262-11e0-8bdf-52540025f9af</a></p> <p>~30K messages "recovery is timed out, evict stale exports" in conf-sanity.test_47.console.client-9-ib.log</p> <p>Several occurrences on Orion with the crazy "recovery is timed out, evict stale exports" flood:</p> <p><a href="https://maloo.whamcloud.com/test_sets/8d84d62e-ce24-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8d84d62e-ce24-11e0-8d02-52540025f9af</a> (8c8e6dc)<br/> <a href="https://maloo.whamcloud.com/test_sets/15d34bc8-cf7e-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/15d34bc8-cf7e-11e0-8d02-52540025f9af</a> (448fc34)</p> <p>Commit: 526c43ec2e47ead878f0df552b74c78b4fc79d1f (Jan 13, 2012)<br/> Maloo: <a href="https://maloo.whamcloud.com/test_sets/6638310e-3f5f-11e1-990e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6638310e-3f5f-11e1-990e-5254004bbbd3</a></p> <p>Another flood.</p> <p>Old ticket for unsupported version</p> Bugzilla ID 22423.0 Development Epic connect ping Rank 1|hzv8mv: Rank (Obsolete) 4933 Severity