Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20856

client background writeback sends OST_SYNC storms

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Since LU-16713, ll_writepages() treats every background writeback pass as memory reclaim (CL_FSYNC_RECLAIM) and sends an OST_SYNC for every stripe of every file it visits, unless the OSC has no uncommitted pages at all. Background writeback runs whenever a client's dirty pages pass the background threshold, which is normal for any client that is writing, and each OST_SYNC forces a journal commit on the OST. At scale this shows as long periods of very high OST_SYNC rates on an OSS, with ost_io queues backing up.

      Reproduced on master with a single client and buffered writes that never call fsync (OST_SYNC per 6 GB written):

      • vm.dirty_background_bytes=8M: 250-760
      • cgroup v2 memory.max=1G on a 6.12 kernel, default dirty settings: 760 within 12 s

      Fix: force a commit from writeback only when it was started for page reclaim; plain background writeback only writes, as it did before LU-16713. With that change the same tests send no OST_SYNC, and throughput is unchanged at default settings.

      sanity test 411d checks this: with background writeback running for the whole of a buffered write that never calls fsync, the client must send far fewer OST_SYNC than OST_WRITE. It fails on master (196 OST_SYNC for 330 OST_WRITE).

      Follow-up work, tracked separately:

      • unstable pages are counted in node NR_WRITEBACK only, not in the page's memcg or the writer's bdi_writeback, so dirty throttling and cgroups do not see them;
      • since kernel v6.0 (e92eebbb0921, also in RHEL 9) wb->dirty_exceeded stays set after writers drop back under their limit, which affects LU-19014's ll_write_end() flush (see LU-19709);
      • reclaim syncs are sent per object, even for writes an earlier sync already committed.

            wc-triage WC Triage
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: