[LU-17768] Allow building rpms with BTF

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Dominique Martinet

Description

Having BTF can be useful for debugging – for example, this allows running commands such as follow accessing function args directly with lustre types:

[root@c9 ~]# bpftrace -e 'kfunc:lnet_md_unlink {
  if ((args->md->md_flags & 0x4 /* aborted */) && args->md->md_refcount) {
    printf("Got a hit (%p, %d, %d): %s\n",
           args->md, args->md->md_flags, args->md->md_refcount, kstack);
  }
}'

(perf can do some impressive tracing with debuginfos but cannot do filtering, systemtap can do this but it requires debuginfos, this works out of the box with only the standard kernel modules packages, at a reasonable size cost (kmod-lustre grows from 32 to 35MB) – otoh, bpftrace cannot do anything useful without BTF as it's not able to get e.g. even just structs from debuginfos...)

 

This requires two things to work:

1/ from a system perspective, the kernel Makefile (scripts/Makefile.modfinal) require that a vmlinux exists in the build tree, so we need something like the following as root:

 

# ln -s /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /usr/src/kernels/$(uname -r)/

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=2273622 to track this in hope of a real solution, rhel/fedora currently only build in BTF for modules part of the kernel build, which isn't lustre friendly... I'm thinking the hard-coded vmlinux path could be a variable so we could point to the vmlinux in /usr/lib/debug directly.

(This obviously needs the kernel-debuginfo installed, and not just the "common" subpackage with ext4 sources for ldiskfs)

2/ from the spec file itself, the debuginfo split command ends up removing the BTF section, this seems to make the strip leave it alone (also posted in the bz for a more streamlined way of doing this, this comes from the kernel spec file):

diff --git a/lustre.spec.in b/lustre.spec.in
index 3a9975884..06b8505ea 100644
--- a/lustre.spec.in
+++ b/lustre.spec.in
@@ -70,6 +70,7 @@
 %{!?kver:    %global kver    %(uname -r)}
 # cut epoch for kmodtool
 %define _kver %(echo %{kver} | sed -e 's/^[0-9]*://')
+%define _kver_re %(echo %kver | sed 's/+/[+]/g')
 # trust version from kdir - but it can lost 'epoch'
 %if %{defined kdir}
        %define _take_kver 1
@@ -172,6 +173,9 @@
                %global requires_kmod_version %{version}_%(echo %{krequires} | sed -r 'y/-/_/; s/^(2\.6\.[0-9]+)_/\\1.0_/;')
        %endif
 %endif
+
+# keep BTF section in modules
+%{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} --keep-section '.BTF' -p '.*\/usr\/src\/kernels/.*|XXX' -o ignored-debuginfo.list -p '/.*/%%{_kver_re}%{?1:[+]%{1}}/.*|/.*%%{_kver_re}%{?1:\+%{1}}(\.debug)?' -o debuginfo%{?1}.list}
 %endif
 
 # RHEL >= 7 comes with systemd

BTF presence can be checked as follow:

$ readelf -S /lib/modules/$(uname -r)/extra/lustre/net/lnet.ko | grep -i BTF
  [51] .BTF              PROGBITS         0000000000000000  000a4540

 

I don't think it's worth pursuing much until the bz gets some action, but at least there'll be "documentation" to build this successfully if someone else cares.

Issue Links

• is related to: LU-8980 Add tracepoint support to Lustre