-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
3
-
9223372036854775807
Lustre clients always set SB_I_CGROUPWB on the superblock (LU-16697, LU-16954). As a result, every inode a cgroup dirties is attached to that cgroup's writeback context.
When a cgroup is removed (for example when a container exits), the kernel has to switch all of those inodes back to a parent writeback context in inode_switch_wbs_work_fn(). On clients with a very large inode cache this produces a large number of CPU-hogging work items that contend on wb->list_lock:
workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us N times, consider switching to WQ_UNBOUND
We have seen this stall the client node's kernel workqueues for minutes, with hung kworkers. During the stall the client stopped pinging, was ping-evicted by all targets, and discarded dirty pages.
Newer kernels reduce the cost of the switching ("writeback: Avoid contention on wb->list_lock when switching inodes" and related patches, v6.18). On older kernels, though, Lustre gives no way to opt out.
Proposal: add client mount options cgroupwb (default, current behaviour) and nocgroupwb, which skips setting SB_I_CGROUPWB so inodes stay on the bdi root writeback context. The trade-off is that dirty throttling becomes system-wide rather than per cgroup (see LU-16713). A patch will follow.