Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12064

Adaptive timeout at_min adjustment & granularity

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • None
    • 9223372036854775807

    Description

      The adaptive timeout code currently works on a granularity of full seconds, and ignores timeouts of "0".  This means the MDS adaptive timeout code doesn't really adjust the timeouts there.

      This means, for example, the bl_ast timeout stays at the default value of 100 seconds * 1.5 (ldlm_bl_timeout), so, 150 seconds.

      This is a very long time to wait, and the AT code is supposed to shorten this.

      There are two obvious approaches here.

      1. Stop ignoring "0" values in the adaptive timeout code, and set a default non-zero at_min (setting it to 1 second should mean no behavioral change, as that's the current minimum real value).  This solution should be simple and shouldn't affect existing installs too much.  (configuring at_min is pretty common anyway)
      2. Update the adaptive timeout code to use more precise time intervals than 1 second.

       

      I'm inclined to #1.  But in real configs, at_min is generally recommended to be something like 40 seconds.  So perhaps we should default to that instead.

       

      Note specifically in the ldlm_bl_timeout we use the max() of this and ldlm_enqueue_min (default is OBD_TIMEOUT_DEFAULT, 100 seconds), so we'll only get down to that value there.

       

      A few open questions here.

      Attachments

        Issue Links

          Activity

            [LU-12064] Adaptive timeout at_min adjustment & granularity

            One proposal that might help here (and in other places) is for the servers to persistently track the maximum number of connected clients, so that the MDS/OSS knows after a restart how many clients might connect and can set at_min to an appropriate value right from the start.

            adilger Andreas Dilger added a comment - One proposal that might help here (and in other places) is for the servers to persistently track the maximum number of connected clients, so that the MDS/OSS knows after a restart how many clients might connect and can set at_min to an appropriate value right from the start.

            The patch that landed is only increasing at_min to a reasonable minimum value. The work to implement a dynamic at_min/at_max based on the number of connected clients has not been done.

            adilger Andreas Dilger added a comment - The patch that landed is only increasing at_min to a reasonable minimum value. The work to implement a dynamic at_min/at_max based on the number of connected clients has not been done.
            pjones Peter Jones added a comment -

            Landed for 2.16

            pjones Peter Jones added a comment - Landed for 2.16

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50609/
            Subject: LU-12064 ptlrpc: set at_min=5 by default
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 46804c2230cc0f72d4472cddd5a37456e1f2fb00

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50609/ Subject: LU-12064 ptlrpc: set at_min=5 by default Project: fs/lustre-release Branch: master Current Patch Set: Commit: 46804c2230cc0f72d4472cddd5a37456e1f2fb00

            "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50609
            Subject: LU-12064 ptlrpc: set at_min=5 by default
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 1805e4dfa8fcba712b5fa3868f26988e7635dbcb

            gerrit Gerrit Updater added a comment - "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50609 Subject: LU-12064 ptlrpc: set at_min=5 by default Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 1805e4dfa8fcba712b5fa3868f26988e7635dbcb

            With ever-increasing core counts on the client, it makes sense to scale the number of "clients" by max_mod_rpcs_in_flight when computing at_min so that a multi-threaded workload on a smaller number of clients is handled similarly to a larger number of clients with max_mod_rpcs_in_flight=1. With the default max_mod_rpcs_in_flight=8 this would be a multiplier of 3 to the calculated at_min value.

            adilger Andreas Dilger added a comment - With ever-increasing core counts on the client, it makes sense to scale the number of "clients" by max_mod_rpcs_in_flight when computing at_min so that a multi-threaded workload on a smaller number of clients is handled similarly to a larger number of clients with max_mod_rpcs_in_flight=1 . With the default max_mod_rpcs_in_flight=8 this would be a multiplier of 3 to the calculated at_min value.

            I think one of the current issues is that at_min=0 allows the ping flood to happen. If there is at_min > 0 it would significantly reduce the flood. Having a reasonable at_min for the cluster size will help significantly.

            adilger Andreas Dilger added a comment - I think one of the current issues is that at_min=0 allows the ping flood to happen. If there is at_min > 0 it would significantly reduce the flood. Having a reasonable at_min for the cluster size will help significantly.

            People

              wc-triage WC Triage
              pfarrell Patrick Farrell (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated: