-
Bug
-
Resolution: Fixed
-
Medium
-
Lustre 2.17.0
-
None
-
3
-
9223372036854775807
Under parallel-scale-nfsv3 (test_connectathon, also seen on iorssf), a Lustre client that re-exports its mount over NFS panics inside a glimpse triggered by knfsd:
kernel BUG at lib/list_debug.c:26! list_add corruption. prev->next should be next (X), but was <high-32-bits-zeroed X>. __list_add_valid.cold __mutex_add_waiter __mutex_lock ll_merge_attr [lustre] cl_glimpse_lock [lustre] __cl_glimpse_size [lustre] ll_getattr_dentry [lustre] svcxdr_encode_post_op_attr [nfsd] nfs3svc_encode_readres [nfsd] nfsd_dispatch / svc_process / nfsd
The crash is intermittent and has been observed across many kernels (5.14.0-503 through 5.14.0-611.x). It is on the Lustre client (llite) side; the interop client/server version difference is not involved even though it frequently hits in interop runs
sample crash: https://testing.whamcloud.com/test_sets/a3e81776-2f20-4ec2-bb37-a0d29724c275
Root cause
ll_getattr_dentry() determines the parent directory of the dentry being stat'd:
parent = dget_parent(de); dir = d_inode(parent); ll_statahead_enter(dir, de);
knfsd operates on disconnected, self-parented dentries obtained from d_obtain_alias() (d_parent points at the dentry itself). For such a dentry dget_parent() returns the file's own dentry, so dir ends up being the regular file itself, not a directory.
The statahead code (ll_statahead_enter(), sa_pattern_fname_detect() ) then updates directory-only members of struct ll_inode_info: lli_sa_match_count, lli_sa_fname_index, .... These share a union with the regular-file members, and lli_sa_match_count overlaps the upper half of lli_size_mutex.wait_list.next. Writing it zeroes the top 32 bits of that list pointer.
The corruption is silent until a later glimpse takes lli_size_mutex on the contended (slow) path: __mutex_add_waiter() walks the corrupted wait_list and hits the list_debug BUG. This is easy to hit under parallel-scale-nfsv3 because many nfsd threads glimpse re-exported files concurrently.
The offending statahead call was added by LU-14361 (regularized fname
statahead pattern) and is active by default (llite.*.enable_statahead_fname=1).
Reproducer
Does not require a full NFS setup — the same disconnected-dentry path can be driven directly on a Lustre mount:
name_to_handle_at() a regular file, then
echo 3 > /proc/sys/vm/drop_caches to evict its dentry;
open_by_handle_at() it (yields the self-parented dentry) and fstat() the fd — this corrupts the file's lli_size_mutex;
with fail_loc=0x1435 (OBD_FAIL_LLITE_STAT_RACE2, holds the size mutex inside ll_merge_attr) set, have several threads fstat() the same fd concurrently — the contending thread reproduces the list_add BUG above.