[LU-11850] Relocating /proc/fs/lustre/ost to /sys/kernel/debug/lustre/ost prevents non-root access

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Upstream
• Affects Version/s: Lustre 2.12.0
• Labels: llnl
• Severity: 3
• Assignee: James A Simmons
• Reporter: Mahmoud Hanafi

Description

For security reasons /sys/kernel/debug is restrict to root only so by relocating /proc/fs/lustre/ost & mdt to /sys/kenrnel/debug/lustre breaks many tools such as 'performance co pilot" that run as non-privilege users. We rely on such tools to collect lustre metric.

We could change the permissions on /sys/kernel/debug but that is not good security practice. Can there be a build option to selected the location?

Issue Links

• is duplicated by: LU-11988 Some lctl *_param values only readable by root
• is related to: LU-11988 Some lctl *_param values only readable by root
• is related to: LU-12244 lfs check subcommand no longer works as non-root user
• is related to: LU-12513 Address YAML limitations
• is related to: LU-17471 Add symlink /proc/fs/lustre/osd-*/*/brw_stats to /sys/kernel/debug/lustre/osd-*/*/brw_stats
• is related to: LU-18709 High frequency performance counters accessible to non-root users
• is related to: LU-12841 Add ability for lctl get_param to get subfield of YAML output
• is related to: LU-16164 Check params and make them valid YAML if they are designed to be
• is related to: LU-17472 Create programming wrappers (python, go) for liblustreapi Netlink stats API
• is related to: LU-8066 Move lustre procfs handling to sysfs and debugfs.
• is related to: LU-9680 Improve the user land to kernel space interface for lustre

Comments

[James A Simmons]

I updated patch 34256. Not perfect but is mostly works now. Give it a try. Note you will need the latest Lustre to make this work.

[James A Simmons]

Hello. Since you brought this up I do have a patch - https://review.whamcloud.com/c/fs/lustre-release/+/34256. After reading this I tracked down the crash I was seeing which was due to a really large message. Stats can be really huge amount of data. I need to adjust the skb. I do need to do more testing for what global_match() can handle.

[Andreas Dilger]

JT, the change of /sys/kernel/debug to root-only happened in the upstream kernel after Lustre started using it, so it would need a kernel patch on all clients (AFAIK), that I don't think anyone wants.

I haven't if there is an easy way to restructure the code back to using /proc/fs/lustre to make these stats available again, but that would probably be the least disruptive code change. The other option would be to add a dedicated "lparamfs" to hold all the Lustre stats so we don't have to deal with the kernel restrictions at all.

[Jean-Thomas Acquaviva]

We would need to provide access to the statistics on the client side from non-root accesses. Such access is mandatory for many performance tools running in user space.

Probably the patch is of modest size, is it possible to have a hot-fix?

[Gerrit Updater]

"James Simmons <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51959
Subject: LU-11850 lov: migrate completely to lu_tgt_descs API
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8e49bf0a866c9214ac72bb85e2c49557615a3dd4

[James A Simmons]

So I have been doing research into the different stat collectors out their. From what I see you can configure them to collect the data from the lustre utilities instead of attempting to read from the debugfs files directly. For example for collectd you would use:

<Plugin exec>

    Exec "myuser:mygroup" "myprog"

   Exec "otheruser" "/path/to/another/binary" "arg0" "arg1"

   NotificationExec "user" "/usr/lib/collectd/exec/handle_notification"

</Plugin>

Looking at LMT and performance co pilot it looks to be the same case. If we can get are utilities to work without root access we should be in good shape.

 

[James A Simmons]

Just pushed a prototype patch which I'm going to use to discsuss Netlink API with other developers. It does sort of work with just md_stats but more is needed.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34256
Subject: LU-11850 obd: use netlink to get lustre stats
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6c74e4ed15ad654b4a20925bd36b8cc0e014d34c

[James A Simmons]

I managed to get the basics working using netlink with obd stats. Just need to figure out how to link into the ptlrpc service.

[James A Simmons]

The kernel has rules about what can be in sysfs. An excellent article covering these rules is here:

https://lwn.net/Articles/378884

Since Lustre has complex data files they are not allowed in sysfs. So the quick fix done for the linux client was moving it to debugfs . The point of this policy was due to proc becoming a dumpster. Now the dumpster is debugfs  Note I have been avoiding the move of several files like stats to debugfs for the OpenSFS tree.

No fear netlink will resolve these issues. I have a prototypes partially working. I just need to work out the nesting of data. I see its the ptlrpc service stats.

[Mahmoud Hanafi]

But why are we considering /sys/kenrel/debug/lustre/ost/... part of "debugging"