Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20462 Lustre Find Utility (LFU)
  3. LU-20730

LFU: llapi_scan_mount() to scan a filesystem's MDTs from a client

XMLWordPrintable

    • Icon: Technical task Technical task
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      What this is

      A client asks every MDT of a mounted filesystem to scan its own objects, and
      the records stream back in bulk. One library call, llapi_scan_mount(), and
      no round trip per object.

      What already exists

      LU-20720 put the OSD object-table scanner in the kernel and exports its Object
      Stream through a character device; LU-20722 consumes that locally as a
      llapi_scan_device() backend. Both run on the server that holds the target.
      Nothing lets a client reach the stream.

      The transport largely existed too. dt_index_walk() drives an index through
      init/load/next/key/key_size/rec/store and the otable iterator implements
      every one, having been built as a dt_index for LFSCK; the container format
      already carries variable-size records under II_FL_VARREC and no keys under
      II_FL_NOKEY; and rp_attrs carries iterator flags from the request to
      init().

      What this adds

      • dt_otable_lfu_rec() in obdclass builds the wire record from
        rec(DORA_ATTR) and rec(DORA_XATTR), so the ring and an RPC carry the
        same bytes from one builder.
      • rec_size() and rec(DORA_LFU) on the ldiskfs and ZFS otable
        iterators, and dt_otable_features, which was declared with no initialiser.
      • The otable FID past dt_index_read()'s whitelist; no dt lock or version
        on an object that is an iterator rather than a file.
      • DOIF_INDEX: a second load() mode that positions at the hash and
        answers as an index does. load()'s existing return cannot change, because
        LFSCK reads a positive value as "the table is over".
      • LL_IOC_LFU_SCAN: llite to the MD export, lmv to the MDT named in it, mdc
        sending OBD_IDX_READ with a bulk of the caller's pages, as
        osp_it_fetch() reads an index across MDTs.
      • llapi_scan_mount() and libscan_client.c: a scan backend whose unit
        of work is an MDT rather than a slice of a target, so the existing worker
        sharding scans N MDTs in parallel. llapi_scan_obj gains
        so_mdt_index, since one target now covers many MDTs and only the record
        knows which answered.
      • llapi_scan_mount.3.

      Measured

      Two nodes, an 88k-file filesystem, -mtime -1 -type f, five alternating
      pairs with caches dropped on both:

      arm                 count   median wall   RPCs to MDT0000
      stock lfs find      87944       8.770 s             88028
      offloaded stream    87958       0.292 s                20
      

      30x on the median and 4401x fewer round trips. The same fixture on one node
      gave 5.394 s for the walk, so a loopback client understates it: the round
      trips it does not pay are the point.

      Correctness

      llapi_scan_mount() against a scan of the targets themselves: 87961 =
      87961 visible and 88051 = 88051 with internal objects, diff 0 both, and the
      same from a client that is not the server. 87957 of those records carry a
      name and parent FID rebuilt from trusted.link.

      Resumes are where an index walk goes wrong, and one RPC exercises none of
      them, so the client buffer was shrunk to force boundaries: ldiskfs 2446 calls
      with 2445 resumes, ZFS 1 through 58 calls, diff 0 at every size.

      What this is not

      It scans the whole filesystem whatever directory names it: a target has no
      notion of a subtree. So it is a filesystem sweep, not a way to look through
      one directory, and the manual page says so. Choosing between it and
      llapi_scan_namespace() is not part of this and is not yet designed.

      Consumers need CAP_SYS_ADMIN. Server-side filtering by user is what the
      HLD defers, and so does this.

      Note on scope

      DORA_LFU and DOIF_INDEX are iterator capabilities and are left in
      LU-20720 rather than moved here, since anything driving the otable could use
      them. They are listed above because this is what they were built for.

            hnishida Hiroshi Nishida
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: