Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20936

llite: allow disabling cgroup writeback (SB_I_CGROUPWB) on client mounts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Lustre clients always set SB_I_CGROUPWB on the superblock (LU-16697, LU-16954). As a result, every inode a cgroup dirties is attached to that cgroup's writeback context.

      When a cgroup is removed (for example when a container exits), the kernel has to switch all of those inodes back to a parent writeback context in inode_switch_wbs_work_fn(). On clients with a very large inode cache this produces a large number of CPU-hogging work items that contend on wb->list_lock:

      workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us N times, consider switching to WQ_UNBOUND
      

      We have seen this stall the client node's kernel workqueues for minutes, with hung kworkers. During the stall the client stopped pinging, was ping-evicted by all targets, and discarded dirty pages.

      Newer kernels reduce the cost of the switching ("writeback: Avoid contention on wb->list_lock when switching inodes" and related patches, v6.18). On older kernels, though, Lustre gives no way to opt out.

      Proposal: add client mount options cgroupwb (default, current behaviour) and nocgroupwb, which skips setting SB_I_CGROUPWB so inodes stay on the bdi root writeback context. The trade-off is that dirty throttling becomes system-wide rather than per cgroup (see LU-16713). A patch will follow.

            Jinshan Jinshan Xiong
            jinshanx Jinshan Xiong
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: