-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Failure
sanity test_160h: @@@@@@ FAIL: trevis-152vm87: expected 2 GC-thread
review-dne-zfs-part-1, RHEL 9.8, 4 MDTs on 2 MDS nodes. Seen on Gerrit 67783 patch set 6. That patch changes GSS code, and this session runs without GSS, so the patch is not involved.
https://testing.whamcloud.com/test_sets/1ff4a8c4-c505-47f1-b9d1-a87d6567709e
What the test expects
The test makes changelog user cl_user2 idle for more than changelog_max_idle_time (10 s). It then sets fail_loc=0x1316 (OBD_FAIL_FORCE_GC_THREAD) on all MDS nodes. This fail_loc makes each new changelog GC thread wait before it purges records. Then it creates files so that each MDT starts a GC thread, unmounts all MDTs in the background, and checks for one waiting GC thread per MDT on each MDS node. trevis-152vm87 holds MDT0000 and MDT0002, so the test expects 2.
What happened
On trevis-152vm87, MDT0000 started a GC thread about 50 ms before the fail_loc was active:
[ 9884.167689] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1316 [ 9884.355308] Lustre: (mdd_dir.c:1079:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 9884.359984] Lustre: (mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl11 idle for 23s with 4 unprocessed records [ 9884.404395] Lustre: *** cfs_fail_loc=1316, val=0***
The message says "starting", not "simulate starting", so the fail_loc check was still false. That thread did not wait. It purged 4 records and exited. The test's own files came less than changelog_min_gc_interval (2 s) later, so MDT0000 did not start a second thread. MDT0000 then unmounted at once, with no thread to wait for:
[ 9886.756604] Lustre: server umount lustre-MDT0000 complete
Only MDT0002's thread was left, so pgrep chlg_gc_thread found 1. The logs do not show which operation wrote the changelog record that started the early GC.
Second problem: no cleanup
The test stops all MDTs and restarts them only at the end of its body. When it fails in the middle, nothing restarts them. Here 160i then failed (lustre-MDT0000: changelog_mask=+hsm failed: 2), 160j failed (mount_client on /mnt/lustre2 failed), and 160k hung until the session timeout (256 minutes).
How often
In the last 94 lustre-reviews runs of sanity test_160h (as of 2026-10-06), 3 failed, all with other messages: 'User cl10 not registered' (LU-15566), and 'user cl11 index expected N, but is N+1'. This message was not in that sample, so it is rare.
Possible fixes (not tested)
- Set changelog_gc=1 only after fail_loc=0x1316 is set. Then no GC thread can start before the fail_loc is active.
- Add a stack_trap that sets fail_loc=0 and restarts any stopped MDT, so a failure here does not break the tests that follow.