Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14146 Massive directory metadata operation performance decrease
  3. LU-6864

DNE3: Support multiple modify RPCs in flight for MDT-MDT connection

Details

    • 9223372036854775807

    Description

      This feature is a complement to the LU-5319 that implements the support of multiple modify RPCs in flight for MDC-MDT connection.

      It will improve the performance of modify metadata cross-MDT operations, while ensuring the correctness of recovery handling (request resend, and MDT recovery).

      Attachments

        1. mdtest-32kb-dom.png
          mdtest-32kb-dom.png
          9 kB
        2. mdtest-32kb-ost.png
          mdtest-32kb-ost.png
          9 kB
        3. mdtest-dir.png
          mdtest-dir.png
          8 kB
        4. mdtest-zero-size.png
          mdtest-zero-size.png
          8 kB

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-14761 DNE2 Metadata degradation

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-16065 replay-single test_81a: rm remote dir failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-16126 Lustre 2.15.51 mdtest fails with MPI_Abort errors while adjusting max_rpcs_in_progress and using large number of clients

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-5319 Support multiple slots per client in last_rcvd file

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Technical task - A technical task. LU-17821 LMR1c: Improve DNE Distributed Transaction Performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-9436 DNE2 - performance improvement with wide stripping directory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-11999 DNE performance improvement

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-6753 Fix several minor improvements to multislots feature

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-12125 Allow parallel rename of regular files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-6864] DNE3: Support multiple modify RPCs in flight for MDT-MDT connection
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/47656/
            Subject: LU-6864 tests: properly skip sanity/245b in interop
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: c4ebdc96061ae9c24ac471b2866f2087bc3e98d4

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/47656/ Subject: LU-6864 tests: properly skip sanity/245b in interop Project: fs/lustre-release Branch: master Current Patch Set: Commit: c4ebdc96061ae9c24ac471b2866f2087bc3e98d4
            gerrit Gerrit Updater added a comment -

            "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47656
            Subject: LU-6864 tests: properly skip sanity/245b in interop
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 91fa883754380b935b92d6e01fe5b1063483cc0c

            gerrit Gerrit Updater added a comment - "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47656 Subject: LU-6864 tests: properly skip sanity/245b in interop Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 91fa883754380b935b92d6e01fe5b1063483cc0c
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/14375/
            Subject: LU-6864 osp: manage number of modify RPCs in flight
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 23028efcae01bf1274a68fd2dd379fbb33300e82

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/14375/ Subject: LU-6864 osp: manage number of modify RPCs in flight Project: fs/lustre-release Branch: master Current Patch Set: Commit: 23028efcae01bf1274a68fd2dd379fbb33300e82
            gerrit Gerrit Updater added a comment -

            "Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/46735
            Subject: LU-6864 osp: manage number of modify RPCs in flight
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: dce4b2800030f981b91249227daee40966ae4afd

            gerrit Gerrit Updater added a comment - "Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/46735 Subject: LU-6864 osp: manage number of modify RPCs in flight Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: dce4b2800030f981b91249227daee40966ae4afd
            adilger Andreas Dilger added a comment -

            James, thanks for testing this out. The graphs would be much easier to compare if they had the before/after results for the same tests on a single graph.

            As for max_mod_rpcs_in_flight for the MDS, I think it is totally reasonable to increase this higher than 8, since we don't want clients to bottleneck on the MDSes if they are busy handling lots of client requests. The MDS has max_rpcs_in_flight=512, so it wouldn't be unreasonable to have the MDS tune this to match mds.MDS.mdt_out.threads_max.

            adilger Andreas Dilger added a comment - James, thanks for testing this out. The graphs would be much easier to compare if they had the before/after results for the same tests on a single graph. As for max_mod_rpcs_in_flight for the MDS, I think it is totally reasonable to increase this higher than 8, since we don't want clients to bottleneck on the MDSes if they are busy handling lots of client requests. The MDS has max_rpcs_in_flight=512 , so it wouldn't be unreasonable to have the MDS tune this to match mds.MDS.mdt_out.threads_max .
            simmonsja James A Simmons added a comment -

            https://www.opensfs.org/wp-content/uploads/Evaluation-of-DoM-SNE-scaling_Simmons_revised051821.pdf

            Looking at the data compared to my LUG talk since it is the same hardware we see improvements in file removals and stats. I do see a regression in reads of small files. I don't know if that is due to recent changes or the patch itself at this point. This is using the new default settings of max_mod_rpcs_in_flight = 8. I need to bump it up to see if I can get more out of it.

            simmonsja James A Simmons added a comment - https://www.opensfs.org/wp-content/uploads/Evaluation-of-DoM-SNE-scaling_Simmons_revised051821.pdf Looking at the data compared to my LUG talk since it is the same hardware we see improvements in file removals and stats. I do see a regression in reads of small files. I don't know if that is due to recent changes or the patch itself at this point. This is using the new default settings of max_mod_rpcs_in_flight = 8. I need to bump it up to see if I can get more out of it.
            simmonsja James A Simmons added a comment -

            The 14375 patch has been revived. Currently I'm collecting data to see if it helps with DNE2 scaling issues.

            simmonsja James A Simmons added a comment - The 14375 patch has been revived. Currently I'm collecting data to see if it helps with DNE2 scaling issues.
            gerrit Gerrit Updater added a comment -

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32412/
            Subject: LU-6864 mdc: move RPC semaphore code to lustre/osp
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 040ca57f2ebd46b3a50bbec279839286ff20bffc

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32412/ Subject: LU-6864 mdc: move RPC semaphore code to lustre/osp Project: fs/lustre-release Branch: master Current Patch Set: Commit: 040ca57f2ebd46b3a50bbec279839286ff20bffc
            adilger Andreas Dilger added a comment -

            Note that the above patch is not a replacement for Grégoire's patch. It is a code cleanup to move the MDC semaphore to the lustre/osp directory, and when http://review.whamcloud.com/14375 is updated that code should probably be removed. In the meantime we avoid allocating the mdc_rpc_lock for each client->MDT connection.

            adilger Andreas Dilger added a comment - Note that the above patch is not a replacement for Grégoire's patch. It is a code cleanup to move the MDC semaphore to the lustre/osp directory, and when http://review.whamcloud.com/14375 is updated that code should probably be removed. In the meantime we avoid allocating the mdc_rpc_lock for each client->MDT connection.
            gerrit Gerrit Updater added a comment -

            Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/32412
            Subject: LU-6864 mdc: move RPC semaphore code to lustre/osp
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 4ba17e20f890b52141bc742500199da3c682c8bd

            gerrit Gerrit Updater added a comment - Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/32412 Subject: LU-6864 mdc: move RPC semaphore code to lustre/osp Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 4ba17e20f890b52141bc742500199da3c682c8bd

            People

              hongchao.zhang Hongchao Zhang
              pichong Gregoire Pichon
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: