[LU-14661] Provide kernel API for adding peer/peer NI

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.15.0
• Assignee: Chris Horn
• Reporter: Chris Horn

Description

Provide kernel API for adding peer and peer NI

Implement LNetAddPeer() and LNetAddPeerNI() APIs to allow other
kernel modules to add peer and peer NIs to LNet.

Peers created via these APIs are not marked as having been configured
by DLC. As such, they can be overwritten by discovery.

Comments

[Alex Zhuravlev]

hornc can we discover new NIDs asynchronously so that the thread processing llog doesn't block?

[Gaurang Tapase]

IMO, increasing timeout isn't the optimial way - mount takes very long (so failover), users have bad experience, etc.

Agree.

[Alex Zhuravlev]

IMO, increasing timeout isn't the optimial way - mount takes very long (so failover), users have bad experience, etc.

[Gaurang Tapase]

We can definitely improve HA agents to wait a bit more (right now they wait for 450s before giving up). I had tried doubling the timeout (900s), still saw the problem. We need a definite timeout before which the mount would succeed.

[Alex Zhuravlev]

the problem that mount takes too long in such a scenario, HA gives up and initiates a failover which can hit the same problem.
would it be possible to make such a "try to connect" async? so mount process (adding nids from the config) is not blocked trying to connect one by one?

[Serguei Smirnov]

I was experimenting with the mount command which lists nids as X1,X2:Y1,Y2 

In my experiment all nids were fake, so all attempts to connect were expected to fail. I was just checking which nids LNet would try to connect to.

Basically, it looks like before LU-14661 patch, LNet would only try to connect to X1 and Y1 when discovering X and Y respectively. With LU-14661 patch, LNet tries to connect to all listed nids. I think this behavior is expected though and is good in the sense that it gives the mount the chance to succeed if X1 is down but X2 is up.

[Alex Zhuravlev]

If so, can you see if https://review.whamcloud.com/#/c/fs/lustre-release/+/47322/ helps?

according to the testing, no, it doesn't help.

[Alex Zhuravlev]

Is LNet peer discovery disabled in the good case but enabled in the bad case?

not quite ready to answer, will have to check with collegue.
this is what we've found yesterday: same branch, same setup - gets stuck with LU-14661 and does well with LU-14661 reverted.

[Chris Horn]

Is LNet peer discovery disabled in the good case but enabled in the bad case?

[Alex Zhuravlev]

Alex Zhuravlev Can you please attach the full log for the bad case as well?

just attached

[Chris Horn]

And can you provide the exact Lustre hash(es) being tested?