Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-13151

Add parent FID to MTIME changelogs

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • Lustre 2.14.0
    • Lustre 2.13.0
    • None
    • 9223372036854775807

    Description

      Some tools that maintain copies of filesystem metadata in an external database monitor changelogs for changes to the filesystem, in order to determine files that need to be rescanned. This can result in a large number of small updates to the external database that can reduce the tool's ingest performance. It might be beneficial to instead track and scan complete directories that contain modified files and update the external database using bulk operations. Adding the parent FID to MTIME changelogs allows to more efficiently determine the parent directories for some types of file data modifications, by issuing OBD_IOC_FID2PATH once for each parent FID, instead of once for each file FID.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-16469 Add parent FID to CL_LAYOUT changelog records

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • In Progress

          Activity

            [LU-13151] Add parent FID to MTIME changelogs
            nangelinas Nikitas Angelinas made changes -
            Link New: This issue is related to LU-16469 [ LU-16469 ]
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.14.0 [ 14490 ]
            Resolution New: Fixed [ 1 ]
            Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
            pjones Peter Jones added a comment -

            Landed for 2.14

            pjones Peter Jones added a comment - Landed for 2.14
            gerrit Gerrit Updater added a comment -

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37264/
            Subject: LU-13151 mdt: add parent FID to Changelog recordss
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: b9d7f8e07ad68a51b4fcc09babe48a68fcbd6041

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37264/ Subject: LU-13151 mdt: add parent FID to Changelog recordss Project: fs/lustre-release Branch: master Current Patch Set: Commit: b9d7f8e07ad68a51b4fcc09babe48a68fcbd6041
            nangelinas Nikitas Angelinas made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]
            nrutman Nathan Rutman added a comment -

            In particular, code like GUFI can take a "suspects" list of modified directories to rescan. Without the parent fid in the changelog, each mtime record needs a fid2path to find the parent directory to rescan. With the parent in the changelog, all the information except the parent can be thrown away, and all records "collapse" into a small set of modified directories.

            nrutman Nathan Rutman added a comment - In particular, code like GUFI can take a "suspects" list of modified directories to rescan. Without the parent fid in the changelog, each mtime record needs a fid2path to find the parent directory to rescan. With the parent in the changelog, all the information except the parent can be thrown away, and all records "collapse" into a small set of modified directories.
            nangelinas Nikitas Angelinas made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Nikitas Angelinas [ nangelinas ]
            nangelinas Nikitas Angelinas added a comment -

            The patch submitted uses the link EA to add the parent FID in the MDT; we have not verified whether this affects performance negatively. We think this patch could help in detecting some types of file data modifications, but not all, as MTIME changelogs don't seem to be generated e.g. when appending to a file.

            nangelinas Nikitas Angelinas added a comment - The patch submitted uses the link EA to add the parent FID in the MDT; we have not verified whether this affects performance negatively. We think this patch could help in detecting some types of file data modifications, but not all, as MTIME changelogs don't seem to be generated e.g. when appending to a file.
            nangelinas Nikitas Angelinas made changes -
            Comment [ A patch for master that uses the link EA to add the parent FID in the MDT is at https://review.whamcloud.com/#/c/37264/. We have not verified whether this affects performance negatively.

            We think this patch could help in detecting some types of file data modifications, but not all, as MTIME changelogs don't seem to be generated e.g. when appending to a file. ]
            gerrit Gerrit Updater added a comment -

            Nikitas Angelinas (nangelinas@cray.com) uploaded a new patch: https://review.whamcloud.com/37264
            Subject: LU-13151 mdt: add parent FID to MTIME changelogs
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 120166579c678d8fa6610b192fc369d447131140

            gerrit Gerrit Updater added a comment - Nikitas Angelinas (nangelinas@cray.com) uploaded a new patch: https://review.whamcloud.com/37264 Subject: LU-13151 mdt: add parent FID to MTIME changelogs Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 120166579c678d8fa6610b192fc369d447131140

            People

              nangelinas Nikitas Angelinas
              nangelinas Nikitas Angelinas
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: