[LU-14062] Setting project ID inheritance on directories can break hardlinking

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.12.5
• Severity: 3
• Assignee: Wang Shilong (Inactive)
• Reporter: Jeff Niles

Description

A colleague of mine mentioned that LU-11872 and how it relates to hardlinks was brought up on the Robinhood mailing list, so we tested some various cases and debugged as much as we could. We were interested in the reported problems, as we were looking to enable project quotas on one of our file systems within the month, but the users of said file system rely fairly heavily on hardlinks. Here's what we came up with.

There are two main issues:
Issue 1: All hardlink project IDs get set to the most recent update to the inode. This is somewhat expected behavior.
Issue 2: You cannot create hardlinks across multiple existing project directories when inheritance is set.

Examples:
Issue 1:

# mkdir hardlink_dir1
# mkdir hardlink_dir2
# touch hardlink_dir1/foo
# cd hardlink_dir2
# ln ../hardlink_dir1/foo ./hardlink_to_fo
# cd ..
# lfs project -srp 1111 hardlink_dir1
# lfs project -d ./hardlink_dir1/foo
 1111 P ./hardlink_dir1/foo
# lfs project -srp 2222 hardlink_dir2
# lfs project -d ./hardlink_dir1/foo
 2222 P ./hardlink_dir1/foo
#

Issue 2:

# mkdir hardlink_dir1
# mkdir hardlink_dir2
# lfs project -srp 1111 hardlink_dir1
# lfs project -srp 2222 hardlink_dir2
# touch hardlink_dir/foo
touch: cannot touch 'hardlink_dir/foo': No such file or directory
# touch hardlink_dir1/foo
# cd hardlink_dir2
# ln ../hardlink_dir1/foo ./hardlink_to_foo
ln: failed to create hard link './hardlink_to_foo' => '../hardlink_dir1/foo': Invalid cross-device link

# cd ..
# lfs project -C -k hardlink_dir2
# lfs project -d hardlink_dir2
 2222 - hardlink_dir2
# cd hardlink_dir2
# ln ../hardlink_dir1/foo ./hardlink_to_foo
# ls -l
total 1
-rw-r--r-- 2 root root 0 Oct 21 14:49 hardlink_to_foo
#

Looking at `lfs_project.c`, it seems that `project_get_xattr()` is using `sys/stat.h`'s `st_mode` to determine if a file meets the `S_ISREG` or `S_ISDIR` test. If it does, it's then processed. This works for symlinks, but the issue is that hardlinks match the `S_ISREG` condition. We thought over some possible ways to resolve this and you could theoretically use something like `st_nlink > 1` to determine if the file is a hardlink, but that seemingly leads to a ton of edge cases. For one, a project ID could be set on the original file, and then if a hardlink were created, you'd no longer be able to update or change the existing project ID. If you decided to programmatically clear the project ID once a hardlink was identified, you'd end up not accounting for any of the data represented by the various hardlinks.

I'm not sure there's a great permanent solution to this, but as of right now if you enable project quotas, hardlinks break in certain circumstances.

Issue Links

• is related to: LU-11872 Request for option not to follow symlink when setting project ID