Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20714

llite: memory safety bugs in the foreign symlink upcall path

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • Lustre 2.18.0
    • Lustre 2.16.0, Lustre 2.15.0, Lustre 2.17.0, Lustre 2.18.0
    • None
    • 3
    • 9223372036854775807

      Summary

      The foreign symlink feature (LU-12682) lets a client turn a foreign LOV/LMV EA into a symlink,
      using a binary format description that root installs through

      /sys/fs/lustre/llite/<fs>-<sbi>/foreign_symlink_upcall_info
      

      Neither end of that pipeline validates its input. The store side parses the format without
      checking that an item fits in the buffer; the parse side trusts the values the items carry,
      mis-sizes the destination path, and samples a shared array outside its lock. Six defects in
      total, in one small file, all present since the feature landed.

      Root installs the format, so the first step needs privilege — but root is the intended user of
      this interface, and the consequences land on unprivileged readlink(): a kernel oops, a heap
      buffer overflow write, and a path built from a truncated or wrong destination.

      The defects

      A. Store side: no "item fits in the buffer" check

      foreign_symlink_upcall_info_store() walks the buffer with while (remaining > 0), only
      checking that count is a multiple of sizeof(__u32). It never checks that the item it is
      about to read fits.

      A1. POSLEN_TYPE has no bounds check at all

      Where STRING_TYPE and EOB_TYPE have one. It reads pos (offset 8) and len
      (offset 12), then remaining -= POSLEN_ITEM_SZ. A 4-byte buffer holding only the type word is
      enough: sysfs hands the store method a kmalloc(count + 1) buffer — 5 bytes, a kmalloc-8
      object — so the read is out of bounds and the subtraction underflows remaining to 2^64^ - 12:

      # printf '\3\0\0\0' > .../foreign_symlink_upcall_info
      LustreError: foreign_symlink_upcall_info_store()) lustre: wrong type
        '4294967222' encountered at pos 16 , with 18446744073709551604 remaining
        bytes, in infos buffer returned by foreign symlink upcall
      

      pos 16 is past the end of a 5-byte allocation and the printed remaining is the underflowed
      value. The walk then continues, bounded only by MAX_NB_UPCALL_ITEMS, over an on-stack array.

      A2. The STRING_TYPE guard underflows the same way

      It computes remaining - offsetof(bytestring) - sizeof(type) in size_t — 20 on LP64. Every
      decrement in the loop is a multiple of 4, so remaining can legitimately be 4, 8, 12 or 16
      there, all below 20. The subtraction wraps and the guard passes; item->size was itself read
      out of bounds and is then used for both OBD_ALLOC() and a memcpy() from past the end:

      # printf '\2\0\0\0' > .../foreign_symlink_upcall_info
      LustreError: constant string allocation has failed for constant string of
        size 18316391873768374784
      ------------[ cut here ]------------
      WARNING: CPU: 0 PID: 9329 at mm/page_alloc.c:4566 __alloc_pages+0x1cf/0x250
        __kmalloc_large_node+0x79/0x110 / __kmalloc+0x322/0x440
        foreign_symlink_upcall_info_store+...
      

      Those bytes do not escape to userspace: installing a format requires the loop to reach
      remaining == 0, and a STRING_TYPE item read with remaining < 20 always underflows
      afterwards, so the copy is freed on the error path. This one is an out-of-bounds read and a wild
      walk, not a disclosure channel.

      A3. A format with no EOB_TYPE item is accepted

      A buffer ending exactly on a complete POSLEN_TYPE item leaves remaining == 0 and is
      installed:

      # printf '\3\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0' > .../foreign_symlink_upcall_info
      # echo $?
      0
      

      ll_foreign_symlink_upcall_parse() has two loops that then disagree about where the array ends:
      the first sums items_size over exactly ll_foreign_symlink_upcall_nb_items entries and that
      sum sizes the destination allocation, while the second runs until it sees EOB_TYPE. Every item
      the second consumes past nb_items is memcpy()}}ed into {{*destname past its end — a *heap
      buffer overflow write* reached by an ordinary readlink(). Observed as far as
      ll_foreign_symlink_upcall_parse()) unexpected type '0' found in items, because the memory past
      the array happened to be zero.

      B. Parse side: the item values are never validated

      B4. pos + len wraps in 32 bits

      lfm_length, pos and len are all __u32 (lustre_user.h:965, :3147-3148), so
      the lfm_length < pos + len test is evaluated in 32 bits. pos = 0xfffffff0, len = 0x10 sums
      to 0 and passes for any EA. pos is unsigned, so lfm_value + pos zero-extends and the
      memcpy() reads 4 GiB above the EA. This oopses the client from an unprivileged
      readlink(2):

      BUG: unable to handle page fault for address: ffff94f90a5bf120
      RIP: 0010:memcpy_erms+0x6/0x10   RSI: ffff94f90a5bf120  RCX/RDX: 0x10
       ll_foreign_symlink_upcall_parse+0x1de/0x5c0 [lustre]
       ll_foreign_readlink_internal+0x7c5/0xb10 [lustre]
       ll_foreign_get_link+0x10a/0x200 [lustre]
       vfs_readlink+0xb2/0x120  /  __x64_sys_readlink+0x1a/0x30
      

      The faulting address minus 0xfffffff0 is exactly lfm_value. A crash is the reliable
      outcome rather than a disclosure: to wrap, len must be at least 2^32^ - pos -
      lfm_length, so a short non-faulting copy forces pos up near 2^32^, while a lower pos
      forces an enormous len that faults.

      B5. items_size overflows int

      It is an int, but foreign_symlink_alloc_and_copy_prefix() takes a size_t. A
      STRING_TYPE item of size 3000 plus a POSLEN_TYPE with len = 0xFFFFF447 sums to exactly
      0xFFFFFFFF, i.e. -1, which becomes SIZE_MAX; {{full_size = suffix_size + prefix_size +
      3}} wraps to 7, sails past the full_size > PATH_MAX test, and OBD_ALLOC() returns 7
      bytes. Measured on an unpatched build: it gets past the allocation and only fails later, in the
      lfm_length check.

      The error path also frees one byte less than was allocated, so each failed parse drifts
      obd_memory up by one and the leak check reports a phantom leak. Measured on the pre-fix build:
      obd_memory ... leaked: 37, "Memory leaks detected".

      B6. A NUL byte in the EA silently truncates the path

      The suffix is memcpy()}}ed verbatim from user-settable foreign EA content. {{readlink() and
      path lookup both stop at the first NUL, so the link resolves to the wrong path while
      lfs getstripe -v prints the whole value.

      C. Use-after-free on the items array

      C7. The items array is sampled outside the read lock

      ll_foreign_symlink_upcall_parse() samples sbi->ll_foreign_symlink_upcall_items in its
      declaration, before taking ll_foreign_symlink_sem for read, then walks that array while
      reading ll_foreign_symlink_upcall_nb_items under the lock. A concurrent write to the parameter
      swaps the array and frees the old one, so the reader can walk freed memory — or run the new count
      over the old, shorter array.

      (Freeing the old array after up_write() is not a bug: down_write() waits for readers
      already inside, so once the pointer is sampled under the read lock no reader can still hold the
      old one. The early sample is the whole bug.)

      Reproducer

      As root on a client, in the mount's namespace (has_same_mount_namespace() gates the store).
      Non-destructive.

      INFO=$(ls -d /sys/fs/lustre/llite/*/foreign_symlink_upcall_info | head -n1)
      
      printf '\3\0\0\0'                          > $INFO   # A1: truncated POSLEN
      printf '\2\0\0\0'                          > $INFO   # A2: truncated STRING
      printf '\2\0\0\0\0\0\0\0\377\377\0\0\0\0\0\0' > $INFO  # A2: memcpy 65535 OOB
      printf '\3\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0'  > $INFO   # A3: accepted, rc=0
      
      # controls that must keep working, before and after the fix:
      # the format l_foreign_symlink installs - POSLEN(0,36) STRING("/") POSLEN(37,36) EOB
      printf '\3\0\0\0\0\0\0\0\0\0\0\0\44\0\0\0\2\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\57\0\0\0\3\0\0\0\0\0\0\0\45\0\0\0\44\0\0\0\1\0\0\0' > $INFO
      # a STRING item whose payload exactly fills the buffer before the terminator
      printf '\2\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0abcd\1\0\0\0' > $INFO
      
      # then, for the parse-side defects, make the parse side run:
      lctl set_param llite/*/foreign_symlink_enable=1
      lfs mkdir --foreign=symlink --xattr="a/b" --flags=0xda05 --mode 0750 /mnt/lustre/d1
      readlink /mnt/lustre/d1
      

      For B4, install POSLEN(0xfffffff0,16) EOB and readlink() a foreign file — that faults the
      node.

            green Oleg Drokin
            green Oleg Drokin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: