Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20837

libcfs: add tracepoints for Lustre debug messages

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      The libcfs debug log (lctl debug / dk) is reliable and always
      available, but it is poorly suited to live analysis. Messages can
      only be read after they reach the trace buffer, filtering is limited
      to debug masks and subsystems, and long runs of continuous logs are
      hard to capture because the buffer wraps or is dropped under memory
      pressure (see LU-15917).

      Add kernel tracepoints (TRACE_EVENT) to libcfs_debug_msg() so that
      external tools (bpftrace, bcc/libbpf, perf, ftrace) can capture,
      filter and process Lustre debug messages at runtime. The existing
      debug log is left as it is. The tracepoints cost almost nothing when
      disabled (a static key check).

      Three tracepoints are added in trace system "lustre_debug":

      • lustre_debug_location: file, function, line, subsystem and mask
        of the message, emitted before formatting
      • lustre_debug_msg: the same fields plus the formatted message text
      • lustre_debug_msg_raw: subsystem, mask, line and message text
        in a ptldebug_header layout

      TP_printk output is YAML/JSON compatible.

      An example bcc tool, contrib/eBPF/lustre_debug_msgs.py, streams
      messages as JSON and filters them by subsystem (-s), mask (-m),
      source file (-f) and message count (-n), e.g.:

      python3 lustre_debug_msgs.py -s S_RPC -m D_ERROR
      

      This is also a building block for exporting Lustre debug events to
      OpenTelemetry pipelines.

            ablagodarenko Artem Blagodarenko
            ablagodarenko Artem Blagodarenko
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: