Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12867

DNE3: new DNE2 hash function to handle rsync temporary filename

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Lustre 2.14.0
    • Labels:
      None
    • Rank (Obsolete):
      9223372036854775807

      Description

      When copying or resyncing a file with rsync it will create a temporary pathname for the target file, copy the data into the temporary pathname, and then rename the file over the original in order to avoid damaging the original file if the rsync is interrupted.

      $ rsync twoshoes.dilger.ca:/myth/tv/2103_20150920012700.mpg .
      ^Z
      ls -l .2*
       3328 -rw-------  1 adilger 2883584 Oct 16 09:10 .2103_20150920012700.mpg.gyCFCs
      

      It is creating a temporary file with a leading "." and a trailing ".XXXXXX" using mktemp(3) or similar. When the temporary filename is renamed to the target filename in a striped directory, it would likely move to a new MDT and become a remote entry. That causes overhead due to invoking remote rename operations, and leaves a lasting performance impact due to the remote directory entry.

      It would be useful to add a new hash function (e.g. FNV_1a_TEMP) that will handle the case of filenames with a leading "." and trailing ".xxxxxx" 6-8 chars after the "." by dropping those parts of the filename and using the remaining characters for the hash, on the assumption that the final filename would be the just the remaining characters. That will keep the rename to be local to a single MDT in most cases, and not be worse than the current case.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                laisiyao Lai Siyao
                Reporter:
                adilger Andreas Dilger
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: