-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
When an ldiskfs target is frozen (for a block-level snapshot: LVM, md, or fsfreeze), osd_object_delete() can still change it. If the object is on the orphan list (oo_on_orphan_list, set by the llog batch block allocation from LU-18562), osd_drop_preallocated_space() calls ldiskfs_truncate(), which starts a journal handle. Nothing checks for the freeze on this path. LU-8071 added freeze protection to osd_trans_create() only.
ext4 does not block a new handle on a frozen filesystem. ext4_freeze() unlocks the journal again after the flush ("we rely on upper layer to stop further updates"), and ext4_journal_check_start() only warns. So the truncate goes ahead on a filesystem whose superblock was just marked clean (needs_recovery cleared).
Seen on
Master 2.17.60_79 plus local snapshot patches (not posted yet), RHEL 9.7 kernel 5.14.0-611.55.1, ldiskfs MDT on an md array. lctl snapshot_create ran 5 times under a create + write load on a client. On the 4th run, kswapd freed the object while the MDT was frozen, 10 ms before the snapshot was taken:
WARNING: CPU: 6 PID: 105 at ldiskfs/ext4_jbd2.c:75 ldiskfs_journal_check_start+0x57/0xa0 [ldiskfs] md/raid:md72: snap fz4 created (id 4, frozen) CPU: 6 PID: 105 Comm: kswapd0 Tainted: G OE ------ --- 5.14.0-611.55.1.el9_7.x86_64 #1 Call Trace: __ldiskfs_journal_start_sb+0x3a/0x190 [ldiskfs] ldiskfs_truncate+0x176/0x4a0 [ldiskfs] osd_drop_preallocated_space+0x2f/0x290 [osd_ldiskfs] osd_object_delete+0x27a/0x320 [osd_ldiskfs] lu_object_free.isra.0+0x93/0x200 [obdclass] lu_site_purge_objects+0x54d/0x770 [obdclass] lu_cache_shrink_scan+0xfc/0x1a0 [obdclass] do_shrink_slab+0x139/0x2f0 shrink_slab+0xa6/0x130
It was seen once in 5 runs with md native snapshots, and not in 5 runs with LVM thin snapshots. It needs memory pressure during the freeze.
Effect
- The WARNING on the console.
- The snapshot may not match its superblock. If the journal commit, or part of the in-place metadata write, reaches the disk before the snapshot is cut, the snapshot says "no recovery needed" but holds a half-applied change. e2fsck on the snapshot would then find errors. Usually nothing reaches the disk in time and the snapshot is fine, but it is not guaranteed.
Fix
Take freeze protection in osd_drop_preallocated_space() around ldiskfs_truncate(): sb_start_intwrite() / sb_end_intwrite(). ext4_evict_inode() does the same for its own truncate, so a thread that frees an inode during a freeze already waits for the thaw today; this only makes our path behave the same way.
Skipping the truncate while frozen is not an option: the inode is on the in-memory orphan list, so it cannot be freed without ldiskfs_truncate() or ldiskfs_orphan_del(), and both need a journal handle.
Reproducer
Single node, master 2.17.60_79, RHEL 9.7. Targets on plain LVM LVs (the MDT is lpvg/mdt0), combined MGS/MDT, formatted and mounted with llmount.sh. Right after the mount, the MGS config llogs written at target registration are cached objects on the orphan list.
dmsetup suspend lpvg-mdt0 # freezes the ldiskfs MDT (freeze_bdev) echo 2 > /proc/sys/vm/drop_caches & # runs lu_cache_shrink_scan() sleep 5; dmsetup resume lpvg-mdt0
Without the fix: WARNING from ldiskfs_journal_check_start() in 2 of 2 runs (1-2 per run), called from drop_caches -> lu_cache_shrink_scan -> osd_object_delete -> osd_drop_preallocated_space -> ldiskfs_truncate. While the MDT is frozen, drop_caches is already inside ldiskfs_truncate(), waiting for a block read from the suspended device. ftrace counted 30-31 calls of osd_drop_preallocated_space() per run.
With the fix (sb_start_intwrite()): no WARNING in 3 of 3 runs, 29-31 calls each. While the MDT is frozen, drop_caches waits in percpu_rwsem_wait() before the truncate, and finishes right after the resume.
The objects are only there after a fresh mount; a second drop_caches on the same mount finds none (0 calls), so each run starts with a new format and mount.
Test
conf-sanity test_174 in the fix patch does the same with the test framework's dm device (FLAKEY=true, the default): it formats and mounts, suspends the MGS device, runs drop_caches and fails if a new ldiskfs_journal_check_start() WARNING appears. Run as ONLY="164 174": FAIL without the fix ("journal handle started on frozen MGS: 0 -> 11"), PASS with it. A patch follows.
- is related to
-
LU-21089 lsnapshot: snapshots of ldiskfs targets on md-kmec arrays
-
- Open
-