[LU-11872] Request for option not to follow symlink when setting project ID

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: New Feature
• Priority: Minor
• Reporter: Mitsuhiro Nishizawa
• Assignee: Wang Shilong (Inactive)
• Resolution: Fixed

Environment

lustre 2.7.21.3-ddn33

Description

Currently, when issuing "lfs project -srp xxxx DIR", command follows symbolic link and set project ID for the target of the link. Customer want a new option not to follow symbolic link.

Can you please think on implementing a new option?

They want to have quota based on directory.
For instance, when there are two users and symlinks are created as follows,

  /home/user01/backup/link01 -> /home/user02/backup/file01
  /home/user02/backup/link02 -> /home/user01/backup/file02

Then, when they issue 'lfs project' command,

  lfs project -p 10001 -s -r /home/user01/backup
  lfs project -p 10002 -s -r /home/user02/backup

Project ID for /home/user01/backup/file02 will be "10002". As they want project IDs for all the files under /home/user01/backup to be "10001", considering this kind of possibility, they need to check all the files under the directory. They just want to simply run a command on a directory.

Issue Links

• is related to: LU-12480 add STATX_PROJID to upstream kernel
• is related to: LU-14062 Setting project ID inheritance on dir...
• is related to: LU-12056 tar doesn't support project id
• is related to: LU-10501 Resource acquired to 'fd' at line 82 ...
• is related to: LU-12947 Project quotas: support of symlinks?
• is related to: LU-16808 lfs find --printf fails on FIFOs and ...
• is related to: LU-11631 symlink migration should not create r...

Comments

[Peter Jones]

wshilong could you please advise as to how much effort it would be to add such a feature in a future version of Lustre?

[Andreas Dilger]

It should be fairly low effort to just skip symlinks when setting the project ID by default. Setting the project ID on the symlinks themselves may be more complex and could be deferred to e2fsck if they are problematic.

[Wang Shilong (Inactive)]

Andreas,

I think problem is we should still be able to change project ID of symlink file as chown could.
Current problem is there is no hook function to handle this in the kernel.

One of simple solution could be we passing parent dir, and symlink inode number in @fsxattr.
and if child inode number is passed in, ioctl will affect to child rather than parents.

What do you think this idea?

[Andreas Dilger]

I think this should be split into two separate patches.

The first patch is for skipping symlinks, since this is easily done and can be backported with little effort since it only affects "lfs". Then an RFC patch for "setfattr" in e2fsprogs, to start a discussion on how this should be fixed properly, with a CC to linux-fsdevel to ask how XFS has solved this problem.

Finally, a second patch for Lustre to set the project ID for symlinks can be implemented once we have some idea how it should be done.

[Wang Shilong (Inactive)]

Hi Andreas,

Fair enough, btw, I think XFS did not solve this problem too..
Let's do this anyway.

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34111
Subject: LU-11872 utils: don't follow link files in default
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 07ff9a3a5a1f333b7e14a90ce991b4f1eea80f57

[Wang Shilong (Inactive)]

There are some discussions flighting on:

1. One way is to recreate symlink which will inherit projectID from parents automatically,
This way works to implement directory Quota, but it doesn't work to set ProjectID of symlink
to a <ID> different from parents, and list symfile's project still not supported.

2. Upstream kernel suggest to add a new flag AT_PROJID and then use fchown_at() to indicate uid is actually projid.

[Wang Shilong (Inactive)]

More questions for upstream suggestion:

For Project Set:

We might need introduce two extra flags here, one flag indicate
use group id to set Project ID.

Another flag means set project Inherit attribute using fchown_at().

How about Get Project?

It looks like there is no extra var for projectID in "struct ksta{",
we could use @attributes to return inherit set or not.

[Wang Shilong (Inactive)]

Some updates:

Send a generic interface to Linux Kernel community for long term goal.
http://patchwork.ozlabs.org/cover/1050205/

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34447
Subject: LU-11872 utils: add option to recreate symlink files
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ed22649195111890d81f258054f17f2b19e8bdcd

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34111/
Subject: LU-11872 utils: don't follow link files in default
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 004b80da5c4b2a7cf4f4885b43c9edec76cd2493

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35392
Subject: LU-11872 utils: don't follow link files in default
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: a326d5792151d1bd956c820ea1c689154a984ec6

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35392/
Subject: LU-11872 utils: don't follow link files in default
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 4cb79268abe8c9405195436e8e8e2ee16577694d

[Jeff Niles]

Edit: realized this should probably be a new issue, as this one is tailored to symlinks. There is similar behavior with hardlinks as well as some additional odd behavior.

Hardlink issue: LU-14062

[Nathan Dauchy (Inactive)]

Can the fix to "recreate symlink files" be moved forward?

The incorrect project ID is preventing migration of directories to new MDTs (DNE1).  Although, this may be revealing a different bug in the "lfs migrate" capability for MDTs getting stuck when project IDs are not fully matching (as hinted at in LU-14062).  Here is a test case:

# lfs project -d .
  0 - .
 # mkdir mdt_migration_test
 # ln -s .. mdt_migration_test/symlink
 # lfs project -p 999 -r -s mdt_migration_test
 lfs: unable to get xattr for symlink 'mdt_migration_test/symlink': Operation not supported
 # lfs migrate -m 3 mdt_migration_test
 mdt_migration_test/symlink migrate failed: Invalid cross-device link (-18)
 # lfs getdirstripe mdt_migration_test
 lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating
 mdtidx FID[seq:oid:ver]
  3 [0x24000ee61:0x3:0x0]
  0 [0x200022085:0x8:0x0]

The only way I have found to get out of the "migrating" state is to clear the project information (so the symlink matches the parent again), redo the migation, and then re-change the project ID. Of course this is quick on a small test case, but for a real directory it could take a long time to walk the tree and change back and forth and the project usage information would be incorrect in the meantime.

# lfs project -C -r mdt_migration_test
lfs: unable to get xattr for symlink 'mdt_migration_test/symlink': Operation not supported
# lfs getdirstripe mdt_migration_test
lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating
mdtidx		 FID[seq:oid:ver]
     3		 [0x24000ee61:0x3:0x0]		
     0		 [0x200022085:0x8:0x0]		
# lfs project -d mdt_migration_test
    0 - mdt_migration_test
# lfs migrate -m 3 mdt_migration_test
# lfs getdirstripe mdt_migration_test
lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none
# lfs project -p 999 -r -s mdt_migration_test
lfs: unable to get xattr for symlink 'mdt_migration_test/symlink': Operation not supported

[Wang Shilong (Inactive)]

I will refresh patch.

[Alexander Zarochentsev]

It should be fairly low effort to just skip symlinks when setting the project ID by default. Setting the project ID on the symlinks themselves may be more complex and could be deferred to e2fsck if they are problematic.

Andreas, are you about not setting/inheriting project id for symlinks while creation? it would be the simplest approach. Inheriting project id from parent while creating a symlink and not having a way to change it later looks not right.

[Andreas Dilger]

Alexander, since symlinks consume inodes, they need to be accounted in the quota, otherwise admins will complain that "find" and the quota count do not match. I think the 34447 patch provides a reasonable solution to the (AFAIK) uncommon case of changing the projid of symlink files, until such a time that the kernel allows this to be done directly.

[Gerrit Updater]

Alexander Zarochentsev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43965
Subject: LU-11872 ldiskfs: do not inherit project id for links
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5f140302f9327afd803149e0081df8068eb7fd2e

[Andreas Dilger]

Alexander, any comment on the resulting inconsistency between project quota inode usage and what "find" would report for the directory, or even that user and group quota will report for the same directory tree?

[Wang Shilong (Inactive)]

I don't think it good idea to "do not inherit project id for links", we tried to introduce a new ioctl for Lustre which passed

Parent dir and child name to get/set project ID, this is even better idea than this IMO..

 

[Alexander Zarochentsev]

>Alexander, any comment on the resulting inconsistency between project quota inode usage and what "find" would report for the directory, or even that user and group quota will report for the same directory tree?

yes it has a drawback. I want to get a feedback whether it is a serious problem or not. https://review.whamcloud.com/#/c/34447 is not updated for about a year, are there plans to move it forward?

[Alexander Zarochentsev]

> we tried to introduce a new ioctl for Lustre which passed
can https://review.whamcloud.com/#/c/34447 be accepted as a temporary solution till the new kernel interface for changing project id is added to the vanilla kernel and that new kernel gets used by RH?

[Wang Shilong (Inactive)]

I recalled when last time i tried to refresh, i was aware of that patch also has a problem even we could recreate symlinks, because there might still some other type of files could have similar problems eg fifo, so it did not fix problem completely though.

 

I could move forward the patch(sorry for delay), even though i prefer idea new ioctl(parent dir and child name), because that could also fix problem get project ID of symlinks/fifo..

 

Any though?

 

[Wang Shilong (Inactive)]

It might take some time to have vanilla kernel to support get/change projectID, i pushed a series patch to upstream to change projectID, and we might extend statx() to get project ID of files, but that might take few years to be RH kernel, so i was still wondering if it ok to a walkaround ioctl in Lustre before that is better to fix problem completely.. rather than continue https://review.whamcloud.com/#/c/34447 patch? 

[Alexander Zarochentsev]

>if it ok to a walkaround ioctl in Lustre before that is better to fix problem completely.. rather than continue https://review.whamcloud.com/#/c/34447 patch.
I think yes.

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44006
Subject: LU-11872 quota: add get/set project support for non-dir/file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b34dff90fc6f8d91640f68ee4cf7acfb70bebe17

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/44006/
Subject: LU-11872 quota: add get/set project support for non-dir/file
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b31792b0e72425c8c7850d69837f08c9f3e95a9c

[Wang Shilong (Inactive)]

Patch landed

[Andreas Dilger]

A possibly better solution instead of adding a specific ioctl for this is to use lgetxattr() and lsetxattr() to get/set the trusted.projid virtual xattr from LU-12056 on symlinks. That will also work for other utilities beyond "lfs project" and also works on ldiskfs-mounted MDTs/OSTs directly.