Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
Lustre 2.9.0
-
None
-
centos7 x86_64
-
9223372036854775807
Description
folks (me included) would like to (continue to) use lustre as a root filesystem in clusters. however CentOS7/RHEL7 uses File Capabilities to set eg. cap_net_admin,cap_net_raw+ep on the 'ping' exe instead of making it setuid root.
llite has long filtered out the security.capability xattr that Capabilities uses. llite currently fakes a return value of success for 'setcap' and returns 'no content' for 'getcap'. This breaks ping, clockdiff, suexec, arping and possibly others.
The behaviour from llite is deliberate and used to be due to stability concerns (b=15587), but stability is unlikely to be a current issue as an enormous amount of work has gone into xattr's since.
Andreas pointed out on lustre-discuss that more recently it's because of performance concerns as eg. 'ls --color' reads security.capability for every file and so more rpc's would be sent. it's expected that the xattr cache (since Lustre 2.5) takes care of some of these concerns as xattr's are cached on clients after the initial accesses.
I made a patch (in gerrit in a sec) that removes the filtering out of security.capability in llite, which I've tested and setcap/getcap and ping etc. now work.
Attachments
Activity
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/27292/
Subject: LU-9562 llite: Remove filtering of seclabel xattr
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6bb6b1ec077a88319cff979bb888cc4d15e62bd0
I'm going to run some benchmarks before and after the patch to see the impact. I think simul and mdtest will give a good view of the impact.
I've been trying to look into Andreas's concerns about rpc's increasing in volume.
in testing lustre 2.9 between a client VM and a mds/oss VM I suspect I'm having troubles counting rpc's sent as there seems to be zero or only small differences in +rpctrace count between llite/xattr.c patched to allow security.capability, and the current llite/xattr.c that filters them out. perhaps rpc's are merged too efficiently to see them? I don't know.
instead I polled /proc/fs/lustre/mdt/-MDT00/exports/*/stats and found the below.
to existing files (dropping client caches before each test) ->
with 'ls color=never' I see 1 getxattr for every 1 getattr.
with 'ls color=always' I see 2 getxattr's for every 1 getattr. 
to non-existent files (dropping client caches before each test) ->
with 'ls color=never' or 'ls color=always' I see 1 getxattr for every 1 getattr, but both in small numbers - only to dirs.
in all four cases, a second run of the test is fully cached on the client - there are 0 getattr's and 0 getxattr's on the MDS, and the lctl dk has no rpc's in it.
the test tree of files is 3 directories deep with 10 files (or non-existent files) in each leaf. eg
only run is any different between the patched xattr.c and the original. with the fake returns from the original llite the result there is as you'd expect - no extra getxattr for security.capability ->
with 'ls color=always' I see 1 getxattr's for every 1 getattr.
cheers,
robin
Robin Humble (plaguedbypenguins@gmail.com) uploaded a new patch: https://review.whamcloud.com/27292
Subject: LU-9562 llite: Remove filtering of seclabel xattr
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b268f9a8390d5dd7ad7464e9f4fda4a8b9e50e12
File creation rate should also be benchmarked, now we have 1 extra RPC on each create so total amount of RPCs is doubled it seems, xattr cache is not helping here. I am not sure about real impact but I expect some. Probably it makes sense to cache empty security.capability xattr for just created files because they have no such xattr for sure.