[LU-12867] DNE3: new DNE2 hash function to handle rsync temporary filename

Related — Thu, 31 Mar 2022 23:45:17 +0000

When copying or resyncing a file with rsync it will create a temporary pathname for the target file, copy the data into the temporary pathname, and then rename the file over the original in order to avoid damaging the original file if the rsync is interrupted.

$ rsync twoshoes.dilger.ca:/myth/tv/2103_20150920012700.mpg .
^Z
ls -l .2*
 3328 -rw-------  1 adilger 2883584 Oct 16 09:10 .2103_20150920012700.mpg.gyCFCs

It is creating a temporary file with a leading "." and a trailing ".XXXXXX" using mktemp(3) or similar. When the temporary filename is renamed to the target filename in a striped directory, it would likely move to a new MDT and become a remote entry. That causes overhead due to invoking remote rename operations, and leaves a lasting performance impact due to the remote directory entry.

It would be useful to add a new hash function (e.g. FNV_1a_TEMP) that will handle the case of filenames with a leading "." and trailing ".xxxxxx" 6-8 chars after the "." by dropping those parts of the filename and using the remaining characters for the hash, on the assumption that the final filename would be the just the remaining characters. That will keep the rename to be local to a single MDT in most cases, and not be worse than the current case.

Whamcloud Community JIRA

[LU-12867] DNE3: new DNE2 hash function to handle rsync temporary filename