Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      Having BTF can be useful for debugging – for example, this allows running commands such as follow accessing function args directly with lustre types:

      [root@c9 ~]# bpftrace -e 'kfunc:lnet_md_unlink {
        if ((args->md->md_flags & 0x4 /* aborted */) && args->md->md_refcount) {
          printf("Got a hit (%p, %d, %d): %s\n",
                 args->md, args->md->md_flags, args->md->md_refcount, kstack);
        }
      }'
      

      (perf can do some impressive tracing with debuginfos but cannot do filtering, systemtap can do this but it requires debuginfos, this works out of the box with only the standard kernel modules packages, at a reasonable size cost (kmod-lustre grows from 32 to 35MB) – otoh, bpftrace cannot do anything useful without BTF as it's not able to get e.g. even just structs from debuginfos...)

       

      This requires two things to work:

      1/ from a system perspective, the kernel Makefile (scripts/Makefile.modfinal) require that a vmlinux exists in the build tree, so we need something like the following as root:

       

      # ln -s /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /usr/src/kernels/$(uname -r)/
      

      I've opened https://bugzilla.redhat.com/show_bug.cgi?id=2273622 to track this in hope of a real solution, rhel/fedora currently only build in BTF for modules part of the kernel build, which isn't lustre friendly... I'm thinking the hard-coded vmlinux path could be a variable so we could point to the vmlinux in /usr/lib/debug directly.

      (This obviously needs the kernel-debuginfo installed, and not just the "common" subpackage with ext4 sources for ldiskfs)

      2/ from the spec file itself, the debuginfo split command ends up removing the BTF section, this seems to make the strip leave it alone (also posted in the bz for a more streamlined way of doing this, this comes from the kernel spec file):

      diff --git a/lustre.spec.in b/lustre.spec.in
      index 3a9975884..06b8505ea 100644
      --- a/lustre.spec.in
      +++ b/lustre.spec.in
      @@ -70,6 +70,7 @@
       %{!?kver:    %global kver    %(uname -r)}
       # cut epoch for kmodtool
       %define _kver %(echo %{kver} | sed -e 's/^[0-9]*://')
      +%define _kver_re %(echo %kver | sed 's/+/[+]/g')
       # trust version from kdir - but it can lost 'epoch'
       %if %{defined kdir}
              %define _take_kver 1
      @@ -172,6 +173,9 @@
                      %global requires_kmod_version %{version}_%(echo %{krequires} | sed -r 'y/-/_/; s/^(2\.6\.[0-9]+)_/\\1.0_/;')
              %endif
       %endif
      +
      +# keep BTF section in modules
      +%{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} --keep-section '.BTF' -p '.*\/usr\/src\/kernels/.*|XXX' -o ignored-debuginfo.list -p '/.*/%%{_kver_re}%{?1:[+]%{1}}/.*|/.*%%{_kver_re}%{?1:\+%{1}}(\.debug)?' -o debuginfo%{?1}.list}
       %endif
       
       # RHEL >= 7 comes with systemd
      

      BTF presence can be checked as follow:

      $ readelf -S /lib/modules/$(uname -r)/extra/lustre/net/lnet.ko | grep -i BTF
        [51] .BTF              PROGBITS         0000000000000000  000a4540

       

      I don't think it's worth pursuing much until the bz gets some action, but at least there'll be "documentation" to build this successfully if someone else cares.

      Attachments

        Issue Links

          Activity

            [LU-17768] Allow building rpms with BTF

            hi! What kernel are you trying this with?

            I just had a look at upstream's kernel and they updated scripts/Makefile.modfinal, it now prefixes vmlinux with `$(objtree)`, so I think we can make this work without the kludge I had requiring a symlink as root on the latest kernels.. (and conversely things might have stopped working with that)
            OTOH the current 5.14.0-503.40.1.el9_5.x86_64 doesn't have this yet so it's perhaps a bit optimistic to think we could rely on this... So given it requires fiddling with the build machine I don't see much benefit from making a real patch out of this, and given my bz has been ignored I don't have much hope redhat will backport that...

            Anyway, I haven't tried recently (built on 5.14.0-362.24.1.el9_3.0.1.x86_64), but I believe it should still work:

            • during the build output, can you check if you had `BTF [M]` messages and "Skipping BTF generation for .* due to unavailability of vmlinux" was not printed?
            • if no (or the message was printed) then you might be missing the kernel debuginfo package or the vmlinux symlink might be bad
            • if yes then strip is the likely culprit and you might want to check the spec file was regenerated from the .in if you applied my diff to the spec.in

            Cheers

            asmadeus Dominique Martinet added a comment - hi! What kernel are you trying this with? I just had a look at upstream's kernel and they updated scripts/Makefile.modfinal, it now prefixes vmlinux with `$(objtree)`, so I think we can make this work without the kludge I had requiring a symlink as root on the latest kernels.. (and conversely things might have stopped working with that) OTOH the current 5.14.0-503.40.1.el9_5.x86_64 doesn't have this yet so it's perhaps a bit optimistic to think we could rely on this... So given it requires fiddling with the build machine I don't see much benefit from making a real patch out of this, and given my bz has been ignored I don't have much hope redhat will backport that... Anyway, I haven't tried recently (built on 5.14.0-362.24.1.el9_3.0.1.x86_64), but I believe it should still work: during the build output, can you check if you had `BTF [M] ` messages and "Skipping BTF generation for .* due to unavailability of vmlinux" was not printed? if no (or the message was printed) then you might be missing the kernel debuginfo package or the vmlinux symlink might be bad if yes then strip is the likely culprit and you might want to check the spec file was regenerated from the .in if you applied my diff to the spec.in Cheers

            I tried your above solution and it didn't work for me. Do you have a patch for this work?

            simmonsja James A Simmons added a comment - I tried your above solution and it didn't work for me. Do you have a patch for this work?

            People

              wc-triage WC Triage
              asmadeus Dominique Martinet
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: