Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20989

lfs find: sleep instead of spinning while the parallel find workers run

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Component: utils (liblustreapi)
      Since: LU-17814 (e505e7dbfb "utils: Add work unit management")

      Current behaviour

      pfind_param_callback() in lustre/utils/liblustreapi_pfind.c waits for the parallel find workers like this:

      	while (ll_atomic_fetch_add(&queue->fwq_active_units, 0) > 0) {
      		/* if a worker hit an error, it forces shutdown... */
      		if (queue->fwq_shutdown)
      			cleanup_work_queue(queue);
      		else
      			sched_yield();
      	}
      

      The workers spend most of their time waiting on the MDS, so sched_yield() returns at once and the main thread spins on a CPU for the whole walk. After a worker sets fwq_shutdown, the loop also spins taking fwq_lock in cleanup_work_queue() until the in-flight units finish.

      This hits every lfs find, not only runs with --threads: calculate_default_thread_count() never returns less than 4, so lfs find always goes through parallel_find(). Other llapi_find() callers with fp_thread_count > 1 are hit too.

      Measured cost

      Single VM (4 vCPU, RHEL 9.7, MGS/MDT/OST/client on one node, 2 MDTs, 2 OSTs), 100 directories x 1000 files. Stock master vs the fix, same lfs binary, library picked with LD_PRELOAD and checked in /proc/<pid>/maps. Arms alternated, 5 or 6 pairs each, MDC lock LRU cleared before each run for "cold". CPU is user+sys from /usr/bin/time; main-thread CPU is from /proc/<pid>/task/<pid>/stat.

      lfs find DIR --threads 4 -mtime -1    (stats every file)
                   wall (median)   process CPU   main thread CPU
      cold stock       4.83 s         4.69 s         1.91 s
      cold fixed       4.08 s         3.26 s         0.00 s
      warm stock       4.86 s         4.62 s         1.88 s
      warm fixed       4.28 s         2.87 s         0.00 s
      
      lfs find DIR --threads 4 -type f      (readdir only, short walk)
      cold stock       0.23 s         0.55 s         0.20 s
      cold fixed       0.24 s         0.34 s         0.00 s
      

      Every pair moved the same way: process CPU -1.2 to -1.9 s and wall -0.5 to -0.8 s per -mtime run. On this node the spinner competes with the workers and the local MDS for the 4 CPUs, so it gets about 40% of a core and slows the walk. On a client with a remote MDS, where the CPUs are idle, we expect it to take a full core (not measured). Output was identical in all runs (100101 lines, same sorted md5).

      Fix

      Add fwq_done_cond to struct find_work_queue. A worker takes fwq_lock for its decrement of fwq_active_units and signals the condvar when it finishes the last unit or after a shutdown. The main thread sleeps on it under the same lock, so the wakeup cannot be lost. On shutdown it drains the queue only when something is queued, then sleeps again. The counter stays atomic because work_unit_create_and_add() still increments it outside the lock.

      struct find_work_queue is in the public lustreapi.h, so the new field changes its size. Inside the tree it is only built on the stack in parallel_find().

      With LLAPI_FAIL_LOC=LLAPI_FAIL_PFIND_SEM the fixed lfs find still exits with "Input/output error" (10 of 10 runs, 0.06 s, no hang). 200 short finds at 2 to 16 threads all finished. Sanity 56* on the fixed build: 87 pass, 0 fail, 1 skip (56xc, needs space), including 56Ek.

            wc-triage WC Triage
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: