Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21234

Lustre delayed work runs on isolated CPUs and stalls DNE lock cancels

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      On servers whose isolcpus cores are held by SCHED_FIFO tasks, DNE lock cancels can hang for many minutes after a large unlink:

      Lustre: ldlm_cn00_007: service thread pid 37083 was inactive for 3660.157 seconds.
       __flush_work+0x26b/0x380
       tgt_sync+0x127/0x150 [ptlrpc]
       tgt_blocking_ast+0x2ac/0x750 [ptlrpc]
      

      Delayed work queued without naming a CPU runs on whichever CPU its timer expires on. On Linux 6.9 to 6.18 that can be an isolated CPU, where the work gets only a few percent of the CPU (fixed in 6.19 by 7dec062cfcf2, not in RHEL 10). tgt_sync() waits for the lu_site_flush pass, so ldlm_cn threads wait too. On master, the PM QoS work, ldlm pool recalc and service watchdog also run on isolated CPUs.

      Reproduced on a single-node DNE VM with isolcpus=2,3 and busy loops on those CPUs: during an rm -rf of 60000 files, a flush pass took 140s and a cross-MDT mkdir 23s.

      Fix: queue delayed work that does not name a CPU on unbound workqueues, which stay off isolated CPUs. With that, the same test ran no Lustre work on the isolated CPUs, and nothing took over 2s.

            wc-triage WC Triage
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: