Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10070

PFL self-extending file layout

Details

    • New Feature
    • Resolution: Fixed
    • Minor
    • Lustre 2.13.0
    • Lustre 2.10.0
    • 9223372036854775807

    Description

      One interesting idea discussed at LAD was to have a PFL layout that is "self extending".

      For several use cases, such as HSM partial-file release/restore, partial-file migration to/from burst buffers, partial-file FLR resync it is advantageous to avoid the need to restore/migrate/resync the entire file at once, but rather only to process the required chunks of the file.

      Essentially, a PFL file would have the normal few components that define the start of the file (e.g. [0-32MB), [32MB-1GB), [1GB-16GB)) and they would be instantiated as with normal PFL today. What is new for "self-extending layouts" is that the last component becomes the template for additional components (as needed) rather than having a component to "EOF" that freezes the layout of the rest of the file.

      This avoids the overhead of explicitly specifying many identical components for the file, only in order to limit the size of the components that need to be processed.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-9096 sanity test_253: File creation failed after rm

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-12681 Data corruption - due incorrect KMS with SEL files

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-12712 sanity-pfl tests triggering “not SEL magic on SEL file”

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-13395 unable to set "--comp-flags=prefer" on a new file

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-13589 PFL "lfs setstripe -E 1M -S 65536" incorrectly parses stripe_size units

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-9846 Overstriping - more than stripe per OST per component

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-10169 Spillover space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-13058 Intermediate component removal (PFL/SEL)

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-9479 sanity test 184d 244: don't instantiate PFL component when taking group lock

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-9809 RTDS(Real-Time Dynamic Striping): A policy based striping framework

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LUDOC-436 Self Extending Layout Documentation

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-8998 Progressive File Layout (PFL)

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-9771 FLR1: Landing tickets for File Level Redundancy Phase 1

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Technical task - A technical task. LU-10808 DoM: component end should align with dom_stripesize

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-7880 add OST/MDT performance statistics to obd_statfs

          • Major - Major loss of function.
          • Open

          Activity

            People

              vitaly_fertman Vitaly Fertman
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: