Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-8207

Add auto-stripe option to lfs_migrate

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • Lustre 2.13.0, Lustre 2.12.4
    • Lustre 2.7.0, Lustre 2.5.3
    • 9223372036854775807

    Description

      The primary reason for this feature is to be able to give users a tool to fix stripe settings on existing files based on file size. Optimal stripe selection for performance is dependent on access patern, but we can make the assumption that large files are more likely to be accessed by multiple clients and benefit from wider striping. The other benefit from restriping is to balance OST capacity used, which an algorithm based solely on file size can more easily address.

      The goal is to have a tool to give the users and say "go restripe your directory with this command" and it will do the right thing in 90% of cases. lfs_migrate already has most of the needed functionality. The change would be to add a "-A" flag to automatically select the stripe count as the file is rewritten. Initial algorithm to determine stripe count is Log2(size_in_GB)+1, though this could change in the future.

      The other application of this patch would be to integrate it with Robinhood (or "lfs find") to generate a list of large, inactive, and narrowly-striped files and pass them to lfs_migrate for restriping. This could be coupled well with OST rebalancing actions.

      For further discussion and context, see mailing list thread "stripe count recommendation, and proposal for auto-stripe tool":
      http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2016-May/013526.html

      Attachments

        Issue Links

          Activity

            [LU-8207] Add auto-stripe option to lfs_migrate
            pjones Peter Jones made changes -
            Labels Original: LTS12 patch New: patch
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.12.4 [ 14690 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-13090 [ LU-13090 ]
            adilger Andreas Dilger made changes -
            Labels Original: patch New: LTS12 patch
            adilger Andreas Dilger made changes -
            Issue Type Original: New Feature [ 2 ] New: Improvement [ 4 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to DDN-848 [ DDN-848 ]
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.13.0 [ 14290 ]
            Resolution New: Fixed [ 1 ]
            Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-5170 [ LU-5170 ]
            utopiabound Nathaniel Clark made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]
            utopiabound Nathaniel Clark made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Nathaniel Clark [ utopiabound ]

            People

              utopiabound Nathaniel Clark
              eschnepp Eric Schnepp
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: