|
Hello, we are still seeing weird behavior with LNet over IB with 2.12+. We have tried to upgrade clients and routers to 2.13 and then 2.14 without success. We went back to 2.12.6 LTS, but we are still seeing occasional kiblnd errors and timeout. The IB fabrics are healthy, sometimes a little bit of congestion but no discard. I'm starting to suspect a deeper problem with LNet/ko2iblnd, where sometimes credits are exhausted? We didn't have with 2.10. To me, the problem seems similar to the one reported in LU-14026 by LLNL.
We do have the following setup:
Fir (serves /scratch) o2ib7 — 8 x lnet routers (IB) — Sherlock v3 (o2ib3)
Last night, one of the 8 routers (sh03-fir06) started to have problems. I've taken traces so that we can investigate.
Router NIDs:
LNet config on the router (we have discovery enabled and using a few Multi-Rail nodes on o2ib3):
Fir Lustre servers on o2ib7 started to exhibit the following errors:
One interesting thing about that router, vs. the 7 others, is that it had a lot of refs (> 3000) in {{/sys/kernel/debug/lnet/nis }} and tx stuck at -367. The high refs count is similar to an issue we noticed with 2.14 routers reported in LU-14584, and we thought that maybe this was a CPT issue. Here, it happened with Lustre 2.12.6 routers.
I took traces on this router at the time of the problem. I'm attaching a zip files sh03-fir06-20210428.zip with the output of:
- lnetctl stats show
- lnetctl peer show
- cat /sys/kernel/debug/lnet/nis
- cat /sys/kernel/debug/lnet/peers
- kernel logs
- short dk logs with +net enabled, just in case that would show something interesting
Rebooting the router fixed the problem.
|
|
Serguei
Could you please advise?
Thanks
Peter
|
|
Hi Stephane,
Could you please provide the output of
from the router and one of the server and one of the clients? It looks like the number of credits configured for the router may be low.
Also, what is the output of
on clients and servers? In the 20 sec captured in the logs there are a few transaction expiring. Perhaps transaction_timeout can be increased, but first let's check the situation with the credits.
Thanks,
Serguei.
|
|
Hi Serguei,
I'm attaching the output of lnetctl net show -v 4:
Output of lnetctl global show:
- on 2.13 clients (1,294 total):
- on 2.12.6 clients (370 total):
- on 2.14 clients (only 6):
As you can see, we are already making sure that transaction_timeout is 50 on all clients and servers, even with 2.13 clients where it was 10 by default. But maybe you will some other problems here? Let me know if you need more data from us. Thanks!
|
|
Stephane,
On servers:
retry_count: 2
transaction_timeout: 50
This means that resulting lnd_timeout is 50/3 seconds, while on other nodes it is 50. If you want to keep the retry_count at 2, please increase transaction_timeout to 150 on the servers, it will make sure every node is using the same lnd_timeout.
Thanks,
Serguei.
|
|
Thanks for catching that, Serguei! Ok, I'll try either that or disable lnet health and set retry_count to 0 like for the other nodes.
|
|
I fixed that issue:
but shortly after that change, another router started to exhibit a similar problem (sh02-fir04). This time it's a router between the same servers (o2ib7, HDR) and o2ib2 (Sherlock v2, another fabric, EDR based), but it's the same idea.
and refs is growing.
On a server:
From a server (10.0.10.115@o2ib7):
Peer show of the server from the router (10.0.10.227@o2ib7/10.50.0.114@o2ib2):
I wonder if there is a way to see what is causing the router to be stuck and accumulating refs. I took another trace with +net on this router while it was stuck. Attaching as sh02-fir04-dknet.log.gz
|
|
Stephane,
From the router net log you provided it looks like the router is not reporting any errors. Was the issue still happening when you were capturing it? I wonder how persistent this issue is, or maybe it just occurs in short bursts. Would it be possible to get net logs from the client and the server, too? That would give some insight into how they qualify failed connections and whether failed connections involve the same router.
"lnetctl net show" output provided for clients and servers indicates that a single ib interface is configured for LNet, are there other ib interfaces on these machines that are not configured for LNet?
|
|
Hi Serguei,
> "lnetctl net show" output provided for clients and servers indicates that a single ib interface is configured for LNet, are there other ib interfaces on these machines that are not configured for LNet?
Not that we know of. Some clients have dual ports card but only one port is up. We do have a few MR clients on o2ib3 though.
Not easy to gather logs on this system (Fir) anymore, as the issue didn't reappear. But this morning, we had a similar issue between another storage system Oak (2.12.6) o2ib5 and Sherlock (o2ib[1-3])
Timeouts and network errors:
I'm attaching this OSS (oak-io2-s1 10.0.2.105@o2ib5) kernel logs as oak-io2-s1.kern.log
Very high refs count on the o2ib5 <> o2ib[1-3] routers :
Peers with queuing from the routers:
I gathered some short +net debug logs from Oak's OSS oak-io2-s1 10.0.2.105@o2ib5, where we can see a drop:
Attaching as oak-io2-s1.dknet9.gz and oak-io2-s1.dknet10.gz (this one shows a Dropping REPLY)
Attaching +net debugs from a client on Sherlock (10.50.14.15@o2ib2), one on o2ib2 I saw some queueing from the routers, but other than that, I picked it randomly and not sure it will be relevant ( sh02-14n15.dknet1.gz sh02-14n15.dknet2.gz sh02-14n15.dknet3.gz )
|
|
Also attaching +net debug logs from client sh02-14n15 10.51.1.14@o2ib3, the one shown above on the OSS with Dropping REPLY. But I think it might have been too late when I gathered the logs. sh03-01n14.dknet.gz
|
|
Stephane,
I still can't see what's going wrong exctly, but for 2.12.6 I'd consider disabling discovery on the routers to avoid extra complexity. For example, it would make sure that the messages sent by a node get to the specific nid of the remote node as decided by the sender, as opposed to the router making this decision when it has discovery enabled. It is more of a general recommendation that may be unrelated to the problem you're seeing though.
If you choose to disable discovery on the routers, you can do so by going one at a time:
- Remove the corresponding route from nodes that are using it (lnetctl route del)
- Disable discovery on the router (if done via the conf file, reload modules)
- Delete the peer representing the router from the nodes that are using it (lnetctl peer del)
- Add the routes back (lnetctl route add)
|
|
Hi Serguei,
We still have this problem with 2.12.7 routers and clients. The servers (Fir) are still running 2.12.5 though. Some routers started to get a high refs count on their NIs and on specific peers, which are generating random RDMA timeouts. If we reboot the routers, that usually doesn't change anything, they become unusable. In some very rare case, (like after requeuing a lot of jobs), we're able to put a router back into production (we were able to do that today), but otherwise, the refs count start to increase immediately.
Clients are on o2ib2, servers on o2ib7. We're pretty much confident the IB fabrics are OK. The problem occurs between routers and servers (o2ib7). Example of a router (at 10.0.10.226@o2ib7) <-> server (OSS) ( at 10.0.10.101@o2ib7) that is stuck:
Debug logs from the router (sh02-fir03 at 10.0.10.226@o2ib7 ) attached as sh02-fir03-debug-lctl_ping_sync.log.gz
Debug logs from the OSS (fir-io1-s1 at 10.0.10.101@o2ib7) at the same time attached as fir-io1-s1_lctl_ping_sync.log.gz
So the router sh02-fir03 (10.0.10.226@o2ib7) is running out of rtr credits, and from the OSS point of view (fir-io1-s1 at 10.0.10.101@o2ib7), it's out of tx credits when this happens, as shown below:
We also noticed that the NIs refs are increasing on the routers when the following messages show up on some OSS's (not all...):
From the OSS, the lnet pinger shows this:
Our problems are with routers 10.0.10.225@o2ib7 and 10.0.10.226@o2ib7 at the moment. Rebooting them doesn't fix the problem.
Our OSS have this configured:
Clients have the default settings (so the difference is check_routers_before_use=0).
Also attaching the peer list of this problematic router (sh02-fir03 at 10.0.10.226@o2ib7) as sh02-fir03.20211020.peers.txt , where we can see that OSS 10.0.10.101@o2ib7 and 10.0.10.102@o2ib7 are out of rtr credits.
Do you know if there a way to identify in the logs what is using these high NI/peers refs count?
|
|
Hi Stephane,
My understanding is that the ref counts go high because the messages on the router queue don't get cleared, same reason as for the negative "available credits". The messages can get stuck on the queue if for some reason the OSS is not responding. After some timeout (50 seconds?) this causes transactions to start expiring. I'll take a closer look at the OSS logs for clues why that may be. We may need to retrieve "ldlm" and "ptlrpc" logs in addition to "net". In the meantime, is there any information about CPU usage on the OSS? Are there any signs of lock-up? Perhaps we can get the output of "echo l > /proc/sysrq-trigger" on the server and the router to check for lock-ups?
Thanks,
Serguei.
|
|
Hi Serguei,
Thanks for this. I checked the OSS for lock-ups but found nothing.
Kernel logs of OSS fir-io1-s2 (10.0.10.102@o2ib7) when we started to put two "bad" routers (10.0.10.225@o2ib7 and 10.0.10.226@o2ib7) back online and problems started again: fir-io1-s2_kernel_sysrq-l_20211026.log.gz (including output of echo l > /proc/sysrq-trigger several times)
Lustre logs on this OSS with dlmtrace and net: fir-io1-s2_dk_net_dlmtrace_20211026.log.gz
Every time we put these two routers back online, the problem shows up with the same two OSS (10.0.10.101@o2ib7 and 0.0.10.102@o2ib7). These OSS are fine with the other routers...
|
|
Adding sh02-fir03-kern_sysrq-l-t_20211026.log.gzwith kernel logs of a router (10.0.10.226@o2ib7) including SysRq l and t dumps, just after the router has been powered on and when the problem happens.
|
|
Hello,
Sorry to interfere here. We observe the same type of issue under network load at the CEA.
Could it be related to LU-15068 ("Race between commit callback and reply_out_callback::LNET_EVENT_SEND")?
|
|
Hi Etienne,
Please interfere! 
It's interesting, and could perhaps match our problem. I also noticed occasional messages like this one on the OSS (here, 10.0.10.225@o2ib7 is one of the bad router):
I'm still wondering why this issue could survive OSS and/or router reboot.
Have you tried a patch for b2_12?
Thanks!
|
|
Hi Etienne and Serguei,
I added the patch from LU-15068 on top of 2.12.7 and we tested it on one OSS and one problematic Lnet router, and unfortunately, that patch doesn't fix this problem. The patch seems to work ok otherwise (no regression noticed).
In this instance, on the router 10.0.10.225@o2ib7, the rtr credits for OSS 10.0.10.102@o2ib7 were quickly dropping way below 0 immediately after putting the router online:
|
|
Hi Stephane,
Just to clarify, was the patch from LU-15068 applied on the 10.0.10.102@o2ib7 in the example above?
Thanks,
Serguei.
|
|
Hi Seirguei,
Yes, we actually started by applying the patch on this OSS (at 10.0.10.102@o2ib7), tested that without success, then we also applied the patch on the router, but hit the same issue, unfortunately.
|
|
Amir and I went over the logs and tried to track down the issue. I don't have a conclusive answer, but here's roughly what appears to be happening, based on the logs provided for the lnetctl ping above:
1) There's a client that stops being responsive. Lnd-level "no credits" messages in the router debug log point to that.
2) If the client is unresponsive, the messages going to it from the server (via router) are not getting finalized. It causes the router credits count for the server to go negative and get stuck at that.
3) The message queue on the server backs up. There are lnd-level "no credits" from ko2iblnd in the server debug log. Normally when the high-watermark level is hit, the node sends a special "NOOP" request bypassing the queue, requesting to release the credits from the other side (the router)
4) If there's no reaction to the "noop", lnd credits are not released and all of them are used up. The server is not able to respond to the lnetctl ping or send any message via the router using this connection.
There are possible causes:
- There's bug with sending the "noop" (server or router)
- There's bug with receiving the "noop" (router or client)
- Nodes are running out of credits legitimately. There's too much to do, they can't keep up.
There are some additional questions/ideas:
At the time when the router debug log was taken, there are actually messages for multiple clients with "no credits" issue. How is the routing configured on the clients? Are all clients able to use all routers or just a subset?
Have you tried increasing the number of credits (go up to peer_credits = 32 peer_credits_hiw = 16 concurrent_sends = 64) ?
Thanks,
Serguei.
|
|
Hi Serguei,
Thanks much for spending time to look at this in more details with Amir. We really appreciate it.
I'm attaching an image to give you an overview LNet architecture of /scratch on Sherlock:
Note: In orange, those are the servers/routers with the repeating credit issues.
There is nothing too fancy here I believe. We have clients on 3 generations of IB fabric (o2ib1, o2ib2 and o2ib3), each connected to Fir (the servers) via their own LNet routers (so o2ib1<>o2ib7, o2ib2<>o2ib7 and o2ib3<>o2ib7). On a single cluster fabric, all compute nodes have the same routing configuration, so yes all clients are using the same routers per fabric. This is a an example from a compute node in o2ib2:
(you can ignore o2ib5 here, which are routers to another storage system)
To answer your second question, no we haven't tried to increase the number of credits. We have been using the default settings. Changing that would mean a full cluster down time and it's ... complicated.
|
|
Hi Serguei,
You wrote:
The message queue on the server backs up. There are lnd-level "no credits" from ko2iblnd in the server debug log. Normally when the high-watermark level is hit, the node sends a special "NOOP" request bypassing the queue, requesting to release the credits from the other side (the router)
Assuming the router or client receives the "NOOP" request, does it release those credits by dropping the messages that were sent but not yet acknowledged (ie freeing the buffers holding those messages, releasing the credits, emitting the LNetError message)?
thanks
|
4) If there's no reaction to the "noop", lnd credits are not released and all of them are used up. The server is not able to respond to the lnetctl ping or send any message via the router using this connection.
I don't understand why the LND doesn't eventually just timeout these transactions when it can't acquire the necessary credits. That would then return the tx credits when those messages are finalized.
|
|
Chris,
I do believe that If noop msg doesn't come so that there are no more credits available to send, eventually there's going to be a timeout for the transaction. The problem may be that during this time the server was not able to talk via the given router to any other node, causing all kinds of backups.
Olaf,
Looks like I got it a bit backwards it my summary. In fact, if the server is sending, it is the receiving side (router), once it figures it accumulated "high-water-mark" or more credits on the receiving end, that is expected to send the "noop" to the server, which, when received by the server, releases the credits to be used again for sending.
Thanks,
Serguei
|
|
It looks like the race that Chris Horn found in LU-15234 could be the cause of our occasional stuck routers and lnet high ref count problems. We look forward to testing the patch when y'all think it's ready on top of 2.12.7.
|
|
Hello! It looks like the patch I mentioned above has been abandoned as it doesn't resolve this issue. I believe it is now our most impactful problem with 2.12 as it seems to start randomly. For example, this morning, we noticed lots of errors of the type on an OSS:
(that's 10.0.10.102@o2ib7)
And checking the ref count on the routers, we saw a very high refcnt on the router (sh02-fir02) for this OSS (10.0.10.102@o2ib7):
Rebooting the router didn't help, its refcnt for 10.0.10.102@o2ib7 is increasing again at this time. Sometimes, rebooting the "bad" router does help though, but unfortunately not always.
Any update on this issue? Thanks!
|
|
Hi Stephane,
There was a couple more patches added for LU-15234: one fixes potential cause of the reference leak, another adds debug information in order to narrow down where the leak may be. The results should be available later this week.
Thanks,
Serguei.
|
|
Hi Serguei,
OK – Thank you!
|
Generated at Sat Feb 10 03:11:36 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
fir-io1-s1_lctl_ping_sync.log.gz
image-2021-11-09-15-37-32-277.png
net_show_client_2.12.txt
sh03-fir06-20210428.zip
