[LU-5570] Better router selection in LNet

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Assignee: Liang Zhen (Inactive)
• Reporter: Liang Zhen (Inactive)

Description

LNet chooses routers based on queued bytes on routers, at the meanwhile, it normally takes tens of seconds to detect dead routers (we see failed completion event of outstanding tx/rx, then close connection and notify LNet peer is dead) , which means it is still possible to queue more messages to a potentially dead router if all other alive routers have long message queue.

we may need to check aliveness timestamp as part of router evaluation, and avoid to choose those routers that are inactive for certain number of seconds as long as there are other active routers (it takes pretty long to mark a router as dead, we might prefer not to choose it before marking it as dead)

Issue Links

• is related to: LU-7734 LNet Multi-Rail Project

Comments

[Gerrit Updater]

Liang Zhen ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13342
Subject: LU-5570 lnet: check router aliveness timestamp
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8650e97e5bede98f3bae16cf64a687ae1c07ef4b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13302/
Subject: Revert "LU-5570 lnet: check router aliveness timestamp"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bfaadd73b74da2aca82007ca78a6baf15ea2790c

[Liang Zhen (Inactive)]

I just requested Oleg to revert it, because this patch is not cleanly rebased, also we need Isaac to review it.

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13302
Subject: Revert "LU-5570 lnet: check router aliveness timestamp"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2d8b8c9e0149b0fe860983cd2020d9781bd2e548

[Jodi Levi (Inactive)]

Patch landed to Master. If there is more work to be done in this ticket, please reopen the ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/11748/
Subject: LU-5570 lnet: check router aliveness timestamp
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 339c7b2b784a528f41c432e9b90285d3445b7536

[Alexey Lyashkov]

it's not true, i have a lots crash dumps with negative credits per destination when router dead.

[Liang Zhen (Inactive)]

In current lnet, if there is any router has positive credit, we will queue message to it not to router with negative credits.

[Alexey Lyashkov]

I think we should don't queue any messages in case negative credits. In that case we will queue only data able to send and easy to put other data to the different routers.

[Liang Zhen (Inactive)]

I would think we should always avoid configuration changes when it's possible, we already have too many tunables which is overkill and very hard for users to make them all corrects.

I totally agree it 's better to fix LU-5485 in this patch (sorry I missed your comment there), and RC ping reduction is also a very good idea, I will have a follow-on patch to implement ping reduction, as it may requires a little more changes to a few different timestamps, so it could be clear to have a separate patch, thanks.

[Isaac Huang (Inactive)]

Also, with aliveness for routers, it'd be possible to fix LU-5485 as well. Better plan them all together.