Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
None
-
9223372036854775807
Description
The adaptive timeout code currently works on a granularity of full seconds, and ignores timeouts of "0". This means the MDS adaptive timeout code doesn't really adjust the timeouts there.
This means, for example, the bl_ast timeout stays at the default value of 100 seconds * 1.5 (ldlm_bl_timeout), so, 150 seconds.
This is a very long time to wait, and the AT code is supposed to shorten this.
There are two obvious approaches here.
- Stop ignoring "0" values in the adaptive timeout code, and set a default non-zero at_min (setting it to 1 second should mean no behavioral change, as that's the current minimum real value). This solution should be simple and shouldn't affect existing installs too much. (configuring at_min is pretty common anyway)
- Update the adaptive timeout code to use more precise time intervals than 1 second.
I'm inclined to #1. But in real configs, at_min is generally recommended to be something like 40 seconds. So perhaps we should default to that instead.
Note specifically in the ldlm_bl_timeout we use the max() of this and ldlm_enqueue_min (default is OBD_TIMEOUT_DEFAULT, 100 seconds), so we'll only get down to that value there.
A few open questions here.
Attachments
Issue Links
Activity
The patch that landed is only increasing at_min to a reasonable minimum value. The work to implement a dynamic at_min/at_max based on the number of connected clients has not been done.
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50609/
Subject: LU-12064 ptlrpc: set at_min=5 by default
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 46804c2230cc0f72d4472cddd5a37456e1f2fb00
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50609
Subject: LU-12064 ptlrpc: set at_min=5 by default
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1805e4dfa8fcba712b5fa3868f26988e7635dbcb
With ever-increasing core counts on the client, it makes sense to scale the number of "clients" by max_mod_rpcs_in_flight when computing at_min so that a multi-threaded workload on a smaller number of clients is handled similarly to a larger number of clients with max_mod_rpcs_in_flight=1. With the default max_mod_rpcs_in_flight=8 this would be a multiplier of 3 to the calculated at_min value.
I think one of the current issues is that at_min=0 allows the ping flood to happen. If there is at_min > 0 it would significantly reduce the flood. Having a reasonable at_min for the cluster size will help significantly.
The main concern about at_min is that if you specify a value inferior to your lnet transaction timeout (modulo the number of lnet retry if it is setup...) the propabilty to flood your lnet networks increases drastically because an rpc not acknowledge by server will be retransmitted by client (not directly but through an high priority rpc) regarding the at_min value. I don't remember by heart the piece of code on client side but it is what we have observed. Anyway in lustre others timeouts like ldlm_timeout, obd_timeout are generally initialized in the code with constant value higher than default transaction timeout. On what we have seen on large scale we could set a default at_min value to lnet_transaction_timeout+1 (for non routing configuration) or max((lnet_transaction_timeout+1),ilog(num_clients) * 3). I think we have to rely on under layers. Does it make sense?
Presentation from CEA set at_min=55 for a cluster with 20,000 clients, so this would also match approximately the ilog(num_clients) * 3 formula (16 * 3 = 48).
What else would be useful here is to tune at_min as a function of the number of clients connected to the servers. For systems with ~200 clients, having at_min=15 is typical, and with ~1500 clients at_min=30 is typical, so a function like the following seems reasonable:
at_min = ilog(num_clients) * 3;
though any explicitly-specified at_min value should take precedence. That probably means having a separate flag that indicates whether at_min has been explicitly set or not, and otherwise recalculating it when clients connect and disconnect.
One proposal that might help here (and in other places) is for the servers to persistently track the maximum number of connected clients, so that the MDS/OSS knows after a restart how many clients might connect and can set at_min to an appropriate value right from the start.