[LU-16106] lnet network NIs go down when they have no peers and check_routers_before_use=1

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.2
• Affects Version/s: Lustre 2.15.0
• Labels: llnl
• Environment: lustre-2.15.0_3.llnl-3.t4.x86_64 ; TOSS 4.4-5
• Severity: 3
• Assignee: Serguei Smirnov
• Reporter: Gian-Carlo Defazio

Description

When starting lnet, we observe NIs going down after about 110 sec. The nodes that we've observed this issue in are router nodes. We setting are setting check_routers_before_use=1 in out lnet module parameters. We do not see this issue with check_routers_before_use=0.

The network is
o2ib18 <{}> tcp129 <{}> o2ib100

The routers of interest are opal[187-190]
opal[187,188] route between o2ib18 and tcp129
opal[189,190] route between tcp129 and o2ib100

o2ib100 includes many non-opal nodes, including LNet routers and MDS and OSS nodes.

The issue was first observed on the tcp network. However, stopping lnet on all nodes in o2ib18 then starting it on opal187 showed the same symptoms on the infiniband NI.

When the NI status was down, traffic was unable to flow between compute nodes on o2ib18 and a filesystem on o2ib100. Also, pings don't work between nodes with down NIs.

However, when starting some nodes opal[188,190] with check_routers_before_use not set, then starting opal[187,189] with check_routers_before_use=1, opal[187,189] are able to ping and be pinged by opal[188,190], but can't ping each other or themselves.

We noticed this when booting the opal cluster. All the non-opal nodes on o2ib100 were up and LNet was running on those non-opal nodes. The opal router nodes listed above were powered on first, and after they were up and LNet was started, the rest of the opal nodes (about 60 lustre clients) were booted. We found that the opal routers NIs were down and the opal clients could not ping through the opal routers to the MDS and OSS nodes on o2ib100. This is a concern because this scenario occurs when we update operating system versions or recover from power outages.

Comments

[Gian-Carlo Defazio]

I tested patch set 3 and it looks good.

Sorry for the late notification, the testing resources with the correct setup for that test have been unavailable lately.

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/48529/
Subject: LU-16106 lnet: allow direct messages regardless of peer NI status
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 9ae1fc3e0e4507c242c5f379e6364ad270d865c0

[Peter Jones]

This fix has now landed for 2.16. We still need to track the port to b2_15 being merged and confirm that LLNL's testing is successful with the latest version

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/48355/
Subject: LU-16106 lnet: allow direct messages regardless of peer NI status
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3345a8a54e89c342a4ce2d8d4bcb04ee919bcd52

[Gian-Carlo Defazio]

We have not tested patch set 3.

[Gerrit Updater]

"Gian-Carlo DeFazio <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48529
Subject: LU-16106 lnet: allow direct messages regardless of peer NI status
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: d747b9c24b9c8366f0551a7b790aad30b3a80786

[Olaf Faaland]

Gian,

Have we tested with the final version of the patch (set 3)?

thanks,

[Gian-Carlo Defazio]

The patch seems to solve the issues we were having. Starting the routers in any order works now. This is with check_routers_before_use=1 of course. The routers can ping each other and compute nodes can get through to routers to the file system.

[Gerrit Updater]

"Serguei Smirnov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48355
Subject: LU-16106 lnet: ignore peer ni down status if it was never up
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 13e7664eef06d081366c7be2e8b43c186f70a429

[Gian-Carlo Defazio]

Serguei,

I've seen the same issue now for our local lustre 2.14 and 2.12 when all other router nodes are down (or have lnet down) and a single node starts lnet. I need to collect more data on what combinations or nodes, startup orders, and module parameters cause the problem. I'll post that info early next week.