[LU-5570] Better router selection in LNet

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Assignee: Liang Zhen (Inactive)
• Reporter: Liang Zhen (Inactive)

Description

LNet chooses routers based on queued bytes on routers, at the meanwhile, it normally takes tens of seconds to detect dead routers (we see failed completion event of outstanding tx/rx, then close connection and notify LNet peer is dead) , which means it is still possible to queue more messages to a potentially dead router if all other alive routers have long message queue.

we may need to check aliveness timestamp as part of router evaluation, and avoid to choose those routers that are inactive for certain number of seconds as long as there are other active routers (it takes pretty long to mark a router as dead, we might prefer not to choose it before marking it as dead)

Issue Links

• is related to: LU-7734 LNet Multi-Rail Project

Comments

[Amir Shehata (Inactive)]

This won't be needed with the Router re-work I have done. LNet Health currently uses the legacy routing code. I reworked the routing code to bring it inline with the Multi-Rail. These changes should resolve the issue described in this ticket.

[Liang Zhen (Inactive)]

James, probably not, I still need your help to sample NI status on router (my last comment on LU-5758)

[James A Simmons]

Do you think this could help with the ARF problems we have been having? Earlier comment seem to point to that.

[Gerrit Updater]

Liang Zhen ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13342
Subject: LU-5570 lnet: check router aliveness timestamp
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8650e97e5bede98f3bae16cf64a687ae1c07ef4b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13302/
Subject: Revert "LU-5570 lnet: check router aliveness timestamp"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bfaadd73b74da2aca82007ca78a6baf15ea2790c

[Liang Zhen (Inactive)]

I just requested Oleg to revert it, because this patch is not cleanly rebased, also we need Isaac to review it.

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13302
Subject: Revert "LU-5570 lnet: check router aliveness timestamp"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2d8b8c9e0149b0fe860983cd2020d9781bd2e548

[Jodi Levi (Inactive)]

Patch landed to Master. If there is more work to be done in this ticket, please reopen the ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/11748/
Subject: LU-5570 lnet: check router aliveness timestamp
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 339c7b2b784a528f41c432e9b90285d3445b7536

[Alexey Lyashkov]

it's not true, i have a lots crash dumps with negative credits per destination when router dead.