-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
Upstream
-
None
-
3
-
9223372036854775807
On kernels before 6.7, shrink_slab() and shrink_slab_memcg() hold shrinker_rwsem for read across each do_shrink_slab() call, and only check whether it is contended between shrinkers. The scan count for the superblock shrinker comes from super_cache_count(), so on a client with a very large dentry/inode cache (tens of millions of Lustre inodes, e.g. after rsync or a policy-engine scan) a single super_cache_scan() pass evicts a huge number of inodes, especially at low reclaim priority (drop_caches, direct or node reclaim).
Evicting a Lustre inode can be slow, because ll_clear_inode() may have to close a cached open handle on the MDS:
mdc_close lmv_close ll_close_inode_openhandle ll_md_real_close ll_clear_inode ll_delete_inode evict dispose_list prune_icache_sb super_cache_scan do_shrink_slab shrink_slab shrink_node __node_reclaim
Meanwhile every down_write(&shrinker_rwsem) waits: registering a shrinker on any mount, and creating a memory cgroup (alloc_shrinker_info()). systemd cannot start services or user sessions, journald hits its watchdog timeout, logins hang, and nodes with hung_task_panic panic.
Reproducer (el9 client): create 10000 files; add a 0.1s delay to ll_delete_inode() with a fail_loc; run echo 2 > /proc/sys/vm/drop_caches in the background; then mkdir /sys/fs/cgroup/<name>. The mkdir blocks until the whole prune has finished (a hung task with hung_task_timeout_secs=22).
Kernels 6.7 and newer are not affected ("mm: shrinker: make global slab shrink lockless", "mm: shrinker: make memcg slab shrink lockless"). RHEL 8 and RHEL 9 clients are. Affected Lustre versions: master and every maintained release (2.15, 2.16, 2.17); the client has never limited the superblock shrinker.
Proposed fix: add a per-mount tunable llite.*.max_shrink (default 0, no limit) that caps the object count the superblock shrinker reports. Each do_shrink_slab() call then scans a bounded number of dentries/inodes, and shrink_slab() can release shrinker_rwsem between calls when someone is waiting for it.
This was seen on HPE customer systems and is fixed in the HPE Lustre branches (HPE ticket LUS-12626).