I think the question we should ask is "should the information revealed in lustre/ost and lustre/mdt be kept away from non-privileged users?" If not, maybe we can reconsider the decision and move the information back to procfs? Or other sysfs location that do not require root privilege to read?
Proc is the worst place for this kind of information. Not only is frowned on by the kernel maintainers but if we every placed a large amount of data into an seq_file, which is used by procfs, the node would be brought to its knees by the cpu load. Actually this has been under discussion on the lustre-devel mailing list. If you are talking about "stats" I'm working a netlink implementation which is way more flexible and has better performance. Give me a few days to roll something out.
BTW here is an excellent article about a solution which I'm modeling my work after.
https://lwn.net/Articles/406975
The article cover why stats in proc is bad.
ok James
We could change the permissions on /sys/kernel/debug but that is not good security practice.
There is no need to change the permissions for the whole of /sys/kernel/debug to be world readable. Currently, it looks like /sys/kernel/debug is itself the only directory that blocks access. It would be possible in the short term to recursively change the permissions of this tree to remove world-readable permissions ("chmod -R go-rw /sys/kernel/debug") and then enable group access permissions for the monitoring tools to the Lustre tree after the filesystem is mounted ("chmod -R g+rX /sys/kernel/debug/lustre; chgrp -R collectd /sys/kernel/debug/lustre" or similar).
James, this is great! Performance is very important! Thanks!
Why is /sys/kernel/debug/lustre not located at /sys/kernel/lustre?
Because all debugfs files go into /sys/kernel/debug. That is the mount point.
But why are we considering /sys/kenrel/debug/lustre/ost/... part of "debugging"
The kernel has rules about what can be in sysfs. An excellent article covering these rules is here:
https://lwn.net/Articles/378884
Since Lustre has complex data files they are not allowed in sysfs. So the quick fix done for the linux client was moving it to debugfs . The point of this policy was due to proc becoming a dumpster. Now the dumpster is debugfs 
Note I have been avoiding the move of several files like stats to debugfs for the OpenSFS tree.
No fear netlink will resolve these issues. I have a prototypes partially working. I just need to work out the nesting of data. I see its the ptlrpc service stats.
I managed to get the basics working using netlink with obd stats. Just need to figure out how to link into the ptlrpc service.
James Simmons (uja.ornl@yahoo.com) uploaded a new patch: https://review.whamcloud.com/34256
Subject: LU-11850 obd: use netlink to get lustre stats
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6c74e4ed15ad654b4a20925bd36b8cc0e014d34c
Just pushed a prototype patch which I'm going to use to discsuss Netlink API with other developers. It does sort of work with just md_stats but more is needed.
So I have been doing research into the different stat collectors out their. From what I see you can configure them to collect the data from the lustre utilities instead of attempting to read from the debugfs files directly. For example for collectd you would use:
<Plugin exec>
Exec "myuser:mygroup" "myprog"
Exec "otheruser" "/path/to/another/binary" "arg0" "arg1"
NotificationExec "user" "/usr/lib/collectd/exec/handle_notification"
</Plugin>
Looking at LMT and performance co pilot it looks to be the same case. If we can get are utilities to work without root access we should be in good shape.
"James Simmons <jsimmons@infradead.org>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51959
Subject: LU-11850 lov: migrate completely to lu_tgt_descs API
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8e49bf0a866c9214ac72bb85e2c49557615a3dd4
We would need to provide access to the statistics on the client side from non-root accesses. Such access is mandatory for many performance tools running in user space.
Probably the patch is of modest size, is it possible to have a hot-fix?
JT, the change of /sys/kernel/debug to root-only happened in the upstream kernel after Lustre started using it, so it would need a kernel patch on all clients (AFAIK), that I don't think anyone wants.
I haven't if there is an easy way to restructure the code back to using /proc/fs/lustre to make these stats available again, but that would probably be the least disruptive code change. The other option would be to add a dedicated "lparamfs" to hold all the Lustre stats so we don't have to deal with the kernel restrictions at all.
Hello. Since you brought this up I do have a patch - https://review.whamcloud.com/c/fs/lustre-release/+/34256. After reading this I tracked down the crash I was seeing which was due to a really large message. Stats can be really huge amount of data. I need to adjust the skb. I do need to do more testing for what global_match() can handle.
I updated patch 34256. Not perfect but is mostly works now. Give it a try. Note you will need the latest Lustre to make this work.
Almost done with the patch for stats. Only bug left is if you grab ALL stats it overflows the liblnetconfig library. I do want to move the internal storage of the stats structures as an Xarray instead of a generic_radix struct.