Details
Question/Request
Minor Description
We recently experienced major issues switching Lustre routing clusters to 2.12 and ended up reverting them to 2.10. In trying to better understand LNet, I read through various documentation pages, but was left with several questions. Can you help answer the following questions and perhaps update the LNet docs as well?
Questions:
- What is the data flow, or order of operations, for sending LNet messages from client to server, through routers? For example, a mock (and incorrect?) model might be:
- Client determines next hop router
- Client checks available routing buffer credits (rtr) on router
- Client checks available send credits (tx) and peer_credits to that router on self
- Client sends <= #peer_credits messages, decrementing tx for each message
- Router receives messages in routing buffers, depending on message size, and decrements # routing buffer credits (rtr) for each message.
- Router then acts as the client, repeating steps 1-5 above to the next hop as well as back to the original client (as data is received)
- Is there any need to manually add peers in a non-MR config? My understanding is no.
- Should a router have a peer entry for every node in the expanded SAN, including in other networks it needs to be routed to?
- The manual states "The number of credits currently in flight (number of transmit credits) is shown in the tx column.... Therefore, rtr – tx is the number of transmits in flight." It seems "in flight" for the "tx" description should be "available" so that rtr-tx would be "in flight", right?
- Should a NID ever show for the wrong interface (e.g. tcp instead of o2ibXX)? We will sometimes see messages in logs from <addr>@tcp when it should be <addr>@o2ibX.
- Do the older mlx4 lnet settings need to be updated for mlx5 or are they still applicable? (https://wiki.lustre.org/LNet_Router_Config_Guide#Configure_Lustre_Servers)?
Attachments
Activity
Thanks Amir. Can you clarify your comment of, "Since o2iblnd is generally more performant than the socklnd, it would make sense to have a larger number of peer_credits for the socklnd network"? Are you saying that because socklnd can't send/recv messages as fast as o2iblnd, you want to increase peer credits to allow more messages to its o2iblnd peers? Wouldn't increasing that number also cause more messages to be send to the socklnd NI, overwhelming it more?
As for credits/buffers, we do have many cases where the min number of RTR or TX credits (via the peers file) show negative, but the buffers via lnetctl routing show are not close to negative. So, rather than a buffer issue, does this imply a higher number of peer_credits (and total credits) needs to be specified?
1) In some cases, we're routing packets from a cluster compute node that may be OPA or IB to a cluster's router node to a data center router node, across a campus WAN, back to another data center router which sits on an IB SAN the Lustre cluster sits on. So, ko2iblnd <> ko2iblnd <> ko2iblnd/ksocklnd <> ksocklnd/ko2iblnd <> ko2iblnd. Does the }}{{credits parameter for the ksocklnd modules need to match the credits parameter for ko2iblnd on router nodes with both interfaces?
2) Given the context in 1), do the number of ko2iblnd credits need to match on servers along the entire path or is it appropriate for router nodes to have a larger number of credits set?
The peer_credits parameters determines how many concurrent messages can be inflight to the same peer. Since o2iblnd is generally more performant than the socklnd, it would make sense to have a larger number of peer_credits for the socklnd network. The o2iblnd negotiates the peer credits per connection. So even if the peer_credits on two nodes are different, they'll be negotiated down to the least common denominator value. I would recommend then to keep the peer_credits the same across homogeneous networks. That said we know of a limitation with socklnd where the performance isn't great per interface. One work around we currently have is to create multiple virtual interfaces on the same ethernet interface and then configure those in MR config. This increases the performance. We're tracking this under: https://jira.whamcloud.com/browse/LU-12815. So this might be a way for you to increase the performance on the socklnd side.
I would also differentiate between the credits module parameters and peer_credits module parameter. The former determines the limit on the total concurrent sends to all peers on a particular Network Interface. While the latter limits the number of concurrent sends per peer. So if you increase the number of peer_credits, you'd want to increase the number of global credits as well for the NI.
The credits are calculated per CPT. You can take a look at {{lnet_ni_tq_credits()}}for more details.
3) Should there be a relation between the number of credits defined for a node's LND driver and it's buffer (or other relevant settings)?
For a router I think you need to look at the total number of large/small/tiny buffers you've specified. lnetctl routing show shows you stats on these buffers; the minimum credits for each. if the minimum credits are dipping in the negative that means you have instances where you're queuing due to lack of buffers. In that case you can increase the buffers allocated for that size. This can be done dynamically via: lnetctl set [tiny_buffers|small_buffers|large_buffers] <value>.
My above comments explain the peer_credits/credits relationship.
Let me know if you have other questions.
Amir - I was trying to keep details of our specific issue out of this documentation ticket. I would like to pass a few more questions past you though:
1) In some cases, we're routing packets from a cluster compute node that may be OPA or IB to a cluster's router node to a data center router node, across a campus WAN, back to another data center router which sits on an IB SAN the Lustre cluster sits on. So, ko2iblnd <> ko2iblnd <> ko2iblnd/ksocklnd <> ksocklnd/ko2iblnd <> ko2iblnd. Does the {{credits}} parameter for the ksocklnd modules need to match the credits parameter for ko2iblnd on router nodes with both interfaces?
2) Given the context in 1), do the number of ko2iblnd credits need to match on servers along the entire path or is it appropriate for router nodes to have a larger number of credits set?
3) Should there be a relation between the number of credits defined for a node's LND driver and it's buffer (or other relevant settings)?
Hi Amir,
I'll create a separate ticket for our issue. It seems like we've worked through the general questions.
In your upgrade procedure do you bring down a router, upgrade to 2.12 and then bring up the router? And that's when you start seeing timeout issues?
Have you tried disabling discovery on the router as you bring it up?
Would we be able to setup a debugging session to get to the bottom of this?
Amir,
We're still having major timeout issues when we try to bring up routers (one by one) in 2.12. Another question I have is whether it should be fine to mix 2.10 and 2.12 routers? I would think so, but wanted to verify.
"On the second-to-last question, the confusion on our part in some cases was because the node should only have had a single NID (@o2ibXX) but the error message referenced @tcp"
There is currently an issue where this could happen. We have fixed it on master.
LU-13477 lnet: Force full discovery cycle
We're currently trying to port it back to b2_12
"can you also clarify the separation of buffers and routing buffers for me? I understand from your flow diagram the regular buffers (divided in bins by page size) are for receiving messages. Are the routing buffers then used once the node (a router) determines the message is not for it, at which point it places the message in a routing buffer to be processed. Is that correct? Are those routing buffers typically larger than the regular buffers?"
For non-gateway nodes (IE clients and servers), usually lustre sets up the RDMA buffer to receive or transmit from. However for gateways, there are no such buffers. In fact lustre doesn't even have to be loaded at all on the gateways. Therefore, a gateway needs to allocate buffers to accept the RDMAed data into and then turn around and forward this data by RDMAing it to the next hop. These are the buffers I was trying to show in the second flow diagram in the link above. In my diagram there are no "regular buffers". I reference only "routing buffers". I'll update the diagram to clarify.
There are 3 router buffer pools, tiny, small and large. These are only allocated when you turn on the routing feature. When a message is received by a gateway and it determines it needs to forward this over to the next hop, then it looks at the size of data in this message and pulls out an appropriately sized buffer (max 1MB). It receives the RDMAed data into that buffer, then turns around and forwards it to the next hop.
Does that answer your question?
Amir, can you also clarify the separation of buffers and routing buffers for me? I understand from your flow diagram the regular buffers (divided in bins by page size) are for receiving messages. Are the routing buffers then used once the node (a router) determines the message is not for it, at which point it places the message in a routing buffer to be processed. Is that correct? Are those routing buffers typically larger than the regular buffers?
Amir,
This is very helpful and answers most of my questions. On the second-to-last question, the confusion on our part in some cases was because the node should only have had a single NID (@o2ibXX) but the error message referenced @tcp. I'll have to recreate the issue and dump lnetctl settings to make sure there isn't a TCP network active on it when we get that message.
Regarding the "Issues" we've been having, we're still trying to characterize them. When we enable Lustre 2.12 on the routers, messages from some clients get jammed up and we start seeing hangs until we take the 2.12 routers offline.
Hi Cameron,
I was thinking from the the router perspective. If a router is forwarding from o2iblnd <->socklnd and you configure the socklnd with more credits than the o2iblnd, you effectively widen the pipe on the socklnd side. So you allow more messages on the socklnd side while throttling the rate on the o2iblnd side. Wouldn't that have the desired effect? And now thinking about it, you can attempt to throttle down the rate by adjusting the max_rpcs_in_flight parameter to reduce the number of RPCs on the o2iblnd side as well. You'd do that on the clients, I believe. Take a look at "39.3.1. Monitoring the Client RPC Stream" in the manual.
RE the credits. When you see credits going negative that means you're starting to queue internally because you're going into throttling mode. That in itself is okay, since you do use the credits to limit the rate to peers. However, if the MIN credits is getting larger and larger (in the negative), that means the system could be getting overwhelmed an is unable to cope with the traffic. In that case I don't think just increasing the credits will be enough. Some other measures will need to be taken. Increasing the number of routers or servers, is an option.
Also note when you start seeing a lot of queuing going on that could eventually lead to timeouts as well. Since a message is tagged by a deadline when you initially send it. The timeout value covers the time messages spend on the queue and the time messages spend on the wire before receiving a completion event (speaking from the o2iblnd perspective). It's a bit of a compound effect. As you get more queuing the time messages wait before they get sent gets longer, which could result in more timeouts. So adjusting the timeout values in this scenario might be needed as well. I'm talking about lnet_transaction_timeout.