[LU-3184] link max_rpcs_in_flight to peer credits

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.5.0
• Type: Bug
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

[Andreas writing with James' account]

The client max_rpcs_in_flight parameter is often tuned to try and get better performance from clients with large bandwidth-delay products. Unfortunately, people are less familiar with tuning the LNET peer credits values, and this results in the Lustre layer trying to send RPCs concurrently that will never go on the wire at the same time.

It would make sense to increase the LNET peer credits when the max_rpcs_in_flight is increased above the current value, so that users get the behaviour that they expect.

Comments

[Isaac Huang (Inactive)]

I don't think they can be linked:

• max_rpcs_in_flight is end-to-end, while LNET peer credits is not. For example, with peer credits 8 and 100 available routers, a client would be able to have at most 800 messages in flight to a server via the routers, in which case you surely don't want to limit max_rpcs_in_flight to be only 8.
• when max_rpcs_in_flight is known, LNET has already initialized and the peer credits can't be changed dynamically.

[Cory Spitz]

You might not be able to tie max_rpcs_in_flight to ni credits either, but it would be easier than peer_credits alone. Maybe this just turns into an LUDOC bug where the guidance on tuning is updated then?

What are your thoughts about linking (in some way, not necessarily rigidly) max_rpcs_in_flight and max_dirty_mb?

[Andreas Dilger]

If you don't think that there should be a link in the code between max_rpcs_in_flight and peer credits/NI credits, could you please submit a patch to the Lustre User Manual to link places that discuss tuning max_rpcs_in_flight to the relevant sections that describe tuning peer credits/NI credits, and how they interact. I still see many places where people advocate increasing max_rpcs_in_flight, but this may not help if there are not enough peer credits.