[LU-11850] Relocating /proc/fs/lustre/ost to /sys/kernel/debug/lustre/ost prevents non-root access

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Upstream
• Affects Version/s: Lustre 2.12.0
• Labels: llnl
• Severity: 3
• Assignee: James A Simmons
• Reporter: Mahmoud Hanafi

Description

For security reasons /sys/kernel/debug is restrict to root only so by relocating /proc/fs/lustre/ost & mdt to /sys/kenrnel/debug/lustre breaks many tools such as 'performance co pilot" that run as non-privilege users. We rely on such tools to collect lustre metric.

We could change the permissions on /sys/kernel/debug but that is not good security practice. Can there be a build option to selected the location?

Issue Links

• is duplicated by: LU-11988 Some lctl *_param values only readable by root
• is related to: LU-11988 Some lctl *_param values only readable by root
• is related to: LU-12244 lfs check subcommand no longer works as non-root user
• is related to: LU-12513 Address YAML limitations
• is related to: LU-17471 Add symlink /proc/fs/lustre/osd-*/*/brw_stats to /sys/kernel/debug/lustre/osd-*/*/brw_stats
• is related to: LU-18709 High frequency performance counters accessible to non-root users
• is related to: LU-12841 Add ability for lctl get_param to get subfield of YAML output
• is related to: LU-16164 Check params and make them valid YAML if they are designed to be
• is related to: LU-17472 Create programming wrappers (python, go) for liblustreapi Netlink stats API
• is related to: LU-8066 Move lustre procfs handling to sysfs and debugfs.
• is related to: LU-9680 Improve the user land to kernel space interface for lustre

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/34256/
Subject: LU-11850 obd: use netlink to get lustre stats
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5803284ac3a5d477df9afffe48ff35f08d67da1a

[Jean-Thomas Acquaviva]

Hi James,

we conducted some tests and it seems that still only root can access the stats. Please find attached a log.

log_stats.txt

[James A Simmons]

Some time numbers for lctl get_param ..stats

root - proc file : 

real    0m0.002s
user    0m0.000s
sys     0m0.002s

normal user (netlink):

real    0m0.006s
user    0m0.003s
sys     0m0.003s

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53994
Subject: LU-11850 obd: debug failure
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ef85e2655bf3d838970c7473c38849330802ec2c

[James A Simmons]

Almost done with the patch for stats. Only bug left is if you grab ALL stats it overflows the liblnetconfig library. I do want to move the internal storage of the stats structures as an Xarray instead of a generic_radix struct.

[James A Simmons]

I updated patch 34256. Not perfect but is mostly works now. Give it a try. Note you will need the latest Lustre to make this work.

[James A Simmons]

Hello. Since you brought this up I do have a patch - https://review.whamcloud.com/c/fs/lustre-release/+/34256. After reading this I tracked down the crash I was seeing which was due to a really large message. Stats can be really huge amount of data. I need to adjust the skb. I do need to do more testing for what global_match() can handle.

[Andreas Dilger]

JT, the change of /sys/kernel/debug to root-only happened in the upstream kernel after Lustre started using it, so it would need a kernel patch on all clients (AFAIK), that I don't think anyone wants.

I haven't if there is an easy way to restructure the code back to using /proc/fs/lustre to make these stats available again, but that would probably be the least disruptive code change. The other option would be to add a dedicated "lparamfs" to hold all the Lustre stats so we don't have to deal with the kernel restrictions at all.

[Jean-Thomas Acquaviva]

We would need to provide access to the statistics on the client side from non-root accesses. Such access is mandatory for many performance tools running in user space.

Probably the patch is of modest size, is it possible to have a hot-fix?

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51959
Subject: LU-11850 lov: migrate completely to lu_tgt_descs API
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8e49bf0a866c9214ac72bb85e2c49557615a3dd4

[James A Simmons]

So I have been doing research into the different stat collectors out their. From what I see you can configure them to collect the data from the lustre utilities instead of attempting to read from the debugfs files directly. For example for collectd you would use:

<Plugin exec>

    Exec "myuser:mygroup" "myprog"

   Exec "otheruser" "/path/to/another/binary" "arg0" "arg1"

   NotificationExec "user" "/usr/lib/collectd/exec/handle_notification"

</Plugin>

Looking at LMT and performance co pilot it looks to be the same case. If we can get are utilities to work without root access we should be in good shape.