Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-6433

MDS deadlock in qouta

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.8.0
    • None
    • 3
    • 9223372036854775807

    Description

      With new quota code MDS obtains quota credits via ptlrpc requests.
      All quota operation takes place after obtaining ldlm lock.
      So, with single MDS it can happen that we are run out of mdt threads to process quota request while all other mdt threads are waiting for a lock conflicting with a lock taken by a thread which is sending quota request.

      The possible solution is to serve quota requests in a HP queue. It looks like quota lock doesn't conflicts, so quota intends can be processed in a HP queue also.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-7926 MDS sits idle with extreme slow response to clients

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Activity

            [LU-6433] MDS deadlock in qouta
            pjones Peter Jones added a comment -

            Landed for 2.8

            pjones Peter Jones added a comment - Landed for 2.8
            gerrit Gerrit Updater added a comment -

            Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/14369/
            Subject: LU-6433 quota: handle QUOTA_DQACQ in READPAGE portal
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 471ad1f679ad7c0193785f82abf6f249ffeb1e79

            gerrit Gerrit Updater added a comment - Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/14369/ Subject: LU-6433 quota: handle QUOTA_DQACQ in READPAGE portal Project: fs/lustre-release Branch: master Current Patch Set: Commit: 471ad1f679ad7c0193785f82abf6f249ffeb1e79
            spitzcor Cory Spitz added a comment -

            Code review (and landing) of http://review.whamcloud.com/14369 was raised to the OpenSFS LWG on 7/1. What blocks this patch from landing?

            spitzcor Cory Spitz added a comment - Code review (and landing) of http://review.whamcloud.com/14369 was raised to the OpenSFS LWG on 7/1. What blocks this patch from landing?
            niu Niu Yawei (Inactive) added a comment -

            Using different portal looks simpler and cleaner to me, actually, I think we can just reuse the READPAGE portal for dqacq request. (or only redirect the synchronous dqacq to the readpage portal).

            niu Niu Yawei (Inactive) added a comment - Using different portal looks simpler and cleaner to me, actually, I think we can just reuse the READPAGE portal for dqacq request. (or only redirect the synchronous dqacq to the readpage portal).
            askulysh Andriy Skulysh added a comment -

            The patch helps because there is always an free HP thread to handle quota request even if all other MDS threads are waiting for a lock. Different portal is more general and complex solution. Is it needed in this case ?

            askulysh Andriy Skulysh added a comment - The patch helps because there is always an free HP thread to handle quota request even if all other MDS threads are waiting for a lock. Different portal is more general and complex solution. Is it needed in this case ?
            niu Niu Yawei (Inactive) added a comment -

            This looks like a livelock problem to me, I don't quite see why adding the quota request to HP list is helpful. I think we usually use different portal (and thread set) to handle such problem, probably we'd introduce a new portal for quota requests?

            niu Niu Yawei (Inactive) added a comment - This looks like a livelock problem to me, I don't quite see why adding the quota request to HP list is helpful. I think we usually use different portal (and thread set) to handle such problem, probably we'd introduce a new portal for quota requests?
            askulysh Andriy Skulysh added a comment -

            patch: http://review.whamcloud.com/14369

            askulysh Andriy Skulysh added a comment - patch: http://review.whamcloud.com/14369
            gerrit Gerrit Updater added a comment -

            Andriy Skulysh (andriy.skulysh@seagate.com) uploaded a new patch: http://review.whamcloud.com/14369
            Subject: LU-6433 quota: handle QUOTA_DQACQ in hp queue
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a4fde3781061b33d660dccb6499469cac6ca1d25

            gerrit Gerrit Updater added a comment - Andriy Skulysh (andriy.skulysh@seagate.com) uploaded a new patch: http://review.whamcloud.com/14369 Subject: LU-6433 quota: handle QUOTA_DQACQ in hp queue Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a4fde3781061b33d660dccb6499469cac6ca1d25

            People

              niu Niu Yawei (Inactive)
              askulysh Andriy Skulysh
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: