-
Improvement
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
The libcfs debug log (lctl debug / dk) is reliable and always
available, but it is poorly suited to live analysis. Messages can
only be read after they reach the trace buffer, filtering is limited
to debug masks and subsystems, and long runs of continuous logs are
hard to capture because the buffer wraps or is dropped under memory
pressure (see LU-15917).
Add kernel tracepoints (TRACE_EVENT) to libcfs_debug_msg() so that
external tools (bpftrace, bcc/libbpf, perf, ftrace) can capture,
filter and process Lustre debug messages at runtime. The existing
debug log is left as it is. The tracepoints cost almost nothing when
disabled (a static key check).
Three tracepoints are added in trace system "lustre_debug":
- lustre_debug_location: file, function, line, subsystem and mask
of the message, emitted before formatting - lustre_debug_msg: the same fields plus the formatted message text
- lustre_debug_msg_raw: subsystem, mask, line and message text
in a ptldebug_header layout
TP_printk output is YAML/JSON compatible.
An example bcc tool, contrib/eBPF/lustre_debug_msgs.py, streams
messages as JSON and filters them by subsystem (-s), mask (-m),
source file (-f) and message count (-n), e.g.:
python3 lustre_debug_msgs.py -s S_RPC -m D_ERROR
This is also a building block for exporting Lustre debug events to
OpenTelemetry pipelines.