Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20462

Lustre Find Utility (LFU)

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Lustre Find Utility (LFU) is a server-side namespace scanning subsystem that replaces slow client-side lfs find traversal with a parallel, structured Object Stream pipeline running directly on MDTs and OSTs.

      Key capabilities:

      • Server-based target scanner (replaces Lester, OSD OI Scrub, Changelog, client-side lfs find)
      • Structured binary Object Stream (FlatBuffers / MsgPack) with pluggable filter rules
      • Parallel multi-MDT scanning with stream merge; MDT-specific indexes (atime, size, mirror status)
      • Zero-copy lockless ring-buffer kernel->userspace export
      • Modular consumer API: feeds FLR resync, PCC-RO, Trash Can, HSM (S3/GCS), tiered storage
      • Open scanner API for third-party tools (RobinHood, PoliMor, GUFI)

      Incremental rollout: server only -> client root -> PtlRPC transport -> regular users

      References:

       

        1.
        		 LFU: llapi namespace scanner API emitting per-object records to a consumer callback Technical task In Progress Hiroshi Nishida
        2.
        		 LFU: reimplement 'lfs find' on the namespace scanner API Technical task In Progress Hiroshi Nishida
        3.
        		 LFU: implement ldiskfs device scanner Technical task In Progress Hiroshi Nishida
        4.
        		 Utils: lfind, the server-side find command for a Lustre target Technical task In Progress Hiroshi Nishida
        5.
        		 llapi: ZFS backend for `llapi_scan_device()` Technical task In Progress Hiroshi Nishida
        6.
        		 MDT-internal objects carry no LMA flag marking them internal Technical task Open Hiroshi Nishida
        7.
        		 LFU: Output Format module to generate pathnames from FIDs Technical task In Progress Hiroshi Nishida
        8.
        		 llapi: read an MDT changelog as a stream of scan records Technical task In Progress Hiroshi Nishida
        9.
        		 LFU: lfs find --since and --changelog, the changelog as a find source Technical task In Progress Hiroshi Nishida
        10.
        		 LFU: fetch attributes with statx for filesystems that are not Lustre Technical task In Progress Hiroshi Nishida

            ablagodarenko Artem Blagodarenko
            ablagodarenko Artem Blagodarenko
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: