https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5570 Lustre <p>LNet chooses routers based on queued bytes on routers, at the meanwhile, it normally takes tens of seconds to detect dead routers (we see failed completion event of outstanding tx/rx, then close connection and notify LNet peer is dead) , which means it is still possible to queue more messages to a potentially dead router if all other alive routers have long message queue.</p> <p>we may need to check aliveness timestamp as part of router evaluation, and avoid to choose those routers that are inactive for certain number of seconds as long as there are other active routers (it takes pretty long to mark a router as dead, we might prefer not to choose it before marking it as dead)</p> LU-5570

Better router selection in LNet

Improvement Minor Resolved Fixed Liang Zhen Liang Zhen Tue, 2 Sep 2014 07:13:29 +0000 Wed, 13 Feb 2019 07:37:13 +0000 Wed, 13 Feb 2019 07:36:58 +0000 0 12 <p>Isaac, Amir, I just posted an experimental patch: <a href="http://review.whamcloud.com/11748" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11748</a><br/> could you take a look and give me some feedback, thanks.</p> <p>How is it different from setting router_ping_timeout to a lower value (and thus detecting dead/congested routers earlier)?</p> <p>if so, then more likely we will drop already queued messages even the router is still alive? I also think It's helpful if user can see last_alive of routers on clients/servers.</p> <p>In case of a false dead router due to low timeout, we don't abandon messages already queued to it, do we? I thought we just stop giving it more messages, and leave those already queued intact.</p> <p>but when we finalise message and return credit, we may drop queued message because router is down? hmm... if without this patch, then this is only happen on router, so you are correct, this is not an issue.<br/> Anyway, I agree changing configuration may prevent upper layer to deliver more messages to potentially dead router, but it may take a few while to recover correct status even it is a false dead (dead_router_check_interval), also, user may not notice that they have to change their configuration. If we have aliveness status for routers, user don't have to change anything?</p> <p>The dead_router_check_interval can be changed at run time, not a big deal really. My point is, the added code complexity seemed to out weight the benefits from the patch. If you want to go forward, why not also avoid unnecessary pings (e.g. no need to ping if last_alive is very recent) - then the additional benefit of reduced pings would make it more worthwhile.</p> <p>Also, with aliveness for routers, it'd be possible to fix <a href="https://jira.whamcloud.com/browse/LU-5485" title="first mount always fail with avoid_asym_router_failure" class="issue-link" data-issue-key="LU-5485"><del>LU-5485</del></a> as well. Better plan them all together.</p> <p>I would think we should always avoid configuration changes when it's possible, we already have too many tunables which is overkill and very hard for users to make them all corrects.</p> <p>I totally agree it 's better to fix <a href="https://jira.whamcloud.com/browse/LU-5485" title="first mount always fail with avoid_asym_router_failure" class="issue-link" data-issue-key="LU-5485"><del>LU-5485</del></a> in this patch (sorry I missed your comment there), and RC ping reduction is also a very good idea, I will have a follow-on patch to implement ping reduction, as it may requires a little more changes to a few different timestamps, so it could be clear to have a separate patch, thanks.</p> <p>I think we should don't queue any messages in case negative credits. In that case we will queue only data able to send and easy to put other data to the different routers.</p> <p>In current lnet, if there is any router has positive credit, we will queue message to it not to router with negative credits.</p> <p>it's not true, i have a lots crash dumps with negative credits per destination when router dead.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/11748/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11748/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5570" title="Better router selection in LNet" class="issue-link" data-issue-key="LU-5570"><del>LU-5570</del></a> lnet: check router aliveness timestamp<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 339c7b2b784a528f41c432e9b90285d3445b7536</p> <p>Patch landed to Master. If there is more work to be done in this ticket, please reopen the ticket.</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13302" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13302</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5570" title="Better router selection in LNet" class="issue-link" data-issue-key="LU-5570"><del>LU-5570</del></a> lnet: check router aliveness timestamp"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2d8b8c9e0149b0fe860983cd2020d9781bd2e548</p> <p>I just requested Oleg to revert it, because this patch is not cleanly rebased, also we need Isaac to review it.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13302/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13302/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5570" title="Better router selection in LNet" class="issue-link" data-issue-key="LU-5570"><del>LU-5570</del></a> lnet: check router aliveness timestamp"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bfaadd73b74da2aca82007ca78a6baf15ea2790c</p> <p>Liang Zhen (liang.zhen@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13342" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13342</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5570" title="Better router selection in LNet" class="issue-link" data-issue-key="LU-5570"><del>LU-5570</del></a> lnet: check router aliveness timestamp<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8650e97e5bede98f3bae16cf64a687ae1c07ef4b</p> <p>Do you think this could help with the ARF problems we have been having? Earlier comment seem to point to that.</p> <p>James, probably not, I still need your help to sample NI status on router (my last comment on <a href="https://jira.whamcloud.com/browse/LU-5758" title="enabling avoid_asym_router_failure prvents the bring up of ORNL production systems" class="issue-link" data-issue-key="LU-5758"><del>LU-5758</del></a>)</p> <p>This won't be needed with the Router re-work I have done. LNet Health currently uses the legacy routing code. I reworked the routing code to bring it inline with the Multi-Rail. These changes should resolve the issue described in this ticket.</p> Related LU-7734 Development Rank 1|hzwv2f: Rank (Obsolete) 15533