Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12569

IBLND_CREDITS_HIGHWATER does not check connection queue depth

Details

    • 3
    • 9223372036854775807

    Description

      The IBLND_CREDITS_HIGHWATER check is used to decide at what # of credits a NOOP message needs to be sent in order to return credits (this is for the case when we send many immediate messages, which consume credits but do not get an acknowledgment, so their credits are not automatically returned).

      However, the check uses a global tunable:

      lnd_peercredits_hiw 

      This tunable is checked against the global peer credits value to make sure it is sane (see kiblnd_tunables_setup()), which is to say that it is less than the total number of credits.

      However, individual connections can have a different queue depth than the global setting (total credits for a connection is equal to the connection queue depth).

      That means if a connection queue depth differs (See particularly kiblnd_create_conn(), the "queue depth reduced" warning message for one case where this can happen) from the global value, it is possible for the highwater mark to be higher than the total number of credits.

      In this case, no NOOP messages will be sent, and it is possible to stall out a connection if both sides send many immediate messages at once.  Essentially, if both ends of a connection send enough immediate messages to exhaust credits, then neither side will send any more messages.  The high water mark is supposed to prevent this by having them send a NOOP before they reach this state.

      But if the highwater mark is greater than the number of credits, this will not occur, and the connection will stall out until a ping or other event causes credits to be returned.

      The solution should be simple - The highwater mark check needs to take in to account the queue depth of an individual connection.

      Attachments

        Issue Links

          Activity

            [LU-12569] IBLND_CREDITS_HIGHWATER does not check connection queue depth
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-15828 [ LU-15828 ]
            pjones Peter Jones made changes -
            Link New: This issue is related to JFC-20 [ JFC-20 ]
            pjones Peter Jones made changes -
            Link Original: This issue is related to JFC-27 [ JFC-27 ]
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.12.3 [ 14418 ]
            Labels Original: LTS12

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36254/
            Subject: LU-12569 o2iblnd: Make credits hiw connection aware
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: 90ba471e367754ea6ddb9a95060591f46b95b0b6

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36254/ Subject: LU-12569 o2iblnd: Make credits hiw connection aware Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 90ba471e367754ea6ddb9a95060591f46b95b0b6

            Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36254
            Subject: LU-12569 o2iblnd: Make credits hiw connection aware
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set: 1
            Commit: 08b218eca6fc01a468b050d6606034b293a8d727

            gerrit Gerrit Updater added a comment - Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36254 Subject: LU-12569 o2iblnd: Make credits hiw connection aware Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 08b218eca6fc01a468b050d6606034b293a8d727
            pjones Peter Jones made changes -
            Link New: This issue is related to JFC-17 [ JFC-17 ]
            pjones Peter Jones made changes -
            Link Original: This issue is related to JFC-19 [ JFC-19 ]
            pjones Peter Jones made changes -
            Link Original: This issue is related to JFC-10 [ JFC-10 ]
            pjones Peter Jones made changes -
            Labels New: LTS12

            People

              ashehata Amir Shehata (Inactive)
              pfarrell Patrick Farrell (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: