Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4639

HSM requests not delivered

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • Lustre 2.6.0, Lustre 2.5.1
    • Lustre 2.5.0
    • 3
    • HSM
    • 12689

    Description

      Issuing too many HSM requests (120 or more, it seems) leads to lnet errors. The corresponding requests, as well as subsequent ones, aren't delivered to the copytool.

      LNetError: 7307:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-0@lo, match 1460297122524484 length 6776 too big: 7600 left, 6144 allowed

      Can be easily reproduced with lfs hsm_archive * against a hundred-ish files.

      Attachments

        Activity

          [LU-4639] HSM requests not delivered
          green Oleg Drokin added a comment -

          The packet drops hang everything because the client will keep retrying indefinitely, as server will not provide any indication it just dropped the request.
          So client is stuck trying to send this rpc with who knows what locks held and blocking other rpcs from being sent

          green Oleg Drokin added a comment - The packet drops hang everything because the client will keep retrying indefinitely, as server will not provide any indication it just dropped the request. So client is stuck trying to send this rpc with who knows what locks held and blocking other rpcs from being sent
          adegremont Aurelien Degremont (Inactive) added a comment -

          Actually, it was not know during HSM design , this was found out later. LNET message size and KUC message size where limitation for big request. I think KUC was patched to detect this. Nothing was done for LNET.
          Anyway, I do not really see why this packet drop will hang everything. We need to work this out.

          adegremont Aurelien Degremont (Inactive) added a comment - Actually, it was not know during HSM design , this was found out later. LNET message size and KUC message size where limitation for big request. I think KUC was patched to detect this. Nothing was done for LNET. Anyway, I do not really see why this packet drop will hang everything. We need to work this out.
          ashehata Amir Shehata (Inactive) added a comment -

          I wasn't involved in the HSM design. But if the limitation was known, what is the expected behavior when this scenario is hit? From the HSM feature perspective, is the message dropped silently at the ptl layer, and that's what's causing the problem?

          ashehata Amir Shehata (Inactive) added a comment - I wasn't involved in the HSM design. But if the limitation was known, what is the expected behavior when this scenario is hit? From the HSM feature perspective, is the message dropped silently at the ptl layer, and that's what's causing the problem?
          adegremont Aurelien Degremont (Inactive) added a comment -

          This limitation was known. But the resulting stale state was not.

          It is a little bit strange that upper layer knows lower layers constraints.

          adegremont Aurelien Degremont (Inactive) added a comment - This limitation was known. But the resulting stale state was not. It is a little bit strange that upper layer knows lower layers constraints.
          ashehata Amir Shehata (Inactive) added a comment -

          as far as I understand, this error message is in lnet_try_match_md(). This is when a message is received and LNet tries to mach a memory descriptor, which was setup to receive these messages. The MD is of a specific size. Now there are some flags that can be set to define what should be done if the message received is larger than the memory descriptor. It can truncate or drop. This particular MD seems to be setup to drop messages.

          I believe either the HSM needs to be aware of this limiation and make sure to send messages of a smaller size, or to increas the size of the MD

          ashehata Amir Shehata (Inactive) added a comment - as far as I understand, this error message is in lnet_try_match_md(). This is when a message is received and LNet tries to mach a memory descriptor, which was setup to receive these messages. The MD is of a specific size. Now there are some flags that can be set to define what should be done if the message received is larger than the memory descriptor. It can truncate or drop. This particular MD seems to be setup to drop messages. I believe either the HSM needs to be aware of this limiation and make sure to send messages of a smaller size, or to increas the size of the MD
          pjones Peter Jones added a comment -

          James

          Please could you set up this reproducer?

          Thanks

          Peter

          pjones Peter Jones added a comment - James Please could you set up this reproducer? Thanks Peter

          People

            jamesanunez James Nunez (Inactive)
            hdoreau Henri Doreau (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: