Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20884

sanity test_160h: 'expected 2 GC-thread' when a GC thread starts before fail_loc is set, then MDTs stay unmounted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Failure

      sanity test_160h: @@@@@@ FAIL: trevis-152vm87: expected 2 GC-thread
      

      review-dne-zfs-part-1, RHEL 9.8, 4 MDTs on 2 MDS nodes. Seen on Gerrit 67783 patch set 6. That patch changes GSS code, and this session runs without GSS, so the patch is not involved.

      https://testing.whamcloud.com/test_sets/1ff4a8c4-c505-47f1-b9d1-a87d6567709e

      What the test expects

      The test makes changelog user cl_user2 idle for more than changelog_max_idle_time (10 s). It then sets fail_loc=0x1316 (OBD_FAIL_FORCE_GC_THREAD) on all MDS nodes. This fail_loc makes each new changelog GC thread wait before it purges records. Then it creates files so that each MDT starts a GC thread, unmounts all MDTs in the background, and checks for one waiting GC thread per MDT on each MDS node. trevis-152vm87 holds MDT0000 and MDT0002, so the test expects 2.

      What happened

      On trevis-152vm87, MDT0000 started a GC thread about 50 ms before the fail_loc was active:

      [ 9884.167689] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1316
      [ 9884.355308] Lustre: (mdd_dir.c:1079:mdd_changelog_store()) lustre-MDD0000:  starting changelog garbage collection
      [ 9884.359984] Lustre: (mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl11 idle for 23s with 4 unprocessed records
      [ 9884.404395] Lustre: *** cfs_fail_loc=1316, val=0***
      

      The message says "starting", not "simulate starting", so the fail_loc check was still false. That thread did not wait. It purged 4 records and exited. The test's own files came less than changelog_min_gc_interval (2 s) later, so MDT0000 did not start a second thread. MDT0000 then unmounted at once, with no thread to wait for:

      [ 9886.756604] Lustre: server umount lustre-MDT0000 complete
      

      Only MDT0002's thread was left, so pgrep chlg_gc_thread found 1. The logs do not show which operation wrote the changelog record that started the early GC.

      Second problem: no cleanup

      The test stops all MDTs and restarts them only at the end of its body. When it fails in the middle, nothing restarts them. Here 160i then failed (lustre-MDT0000: changelog_mask=+hsm failed: 2), 160j failed (mount_client on /mnt/lustre2 failed), and 160k hung until the session timeout (256 minutes).

      How often

      In the last 94 lustre-reviews runs of sanity test_160h (as of 2026-10-06), 3 failed, all with other messages: 'User cl10 not registered' (LU-15566), and 'user cl11 index expected N, but is N+1'. This message was not in that sample, so it is rare.

      Possible fixes (not tested)

      1. Set changelog_gc=1 only after fail_loc=0x1316 is set. Then no GC thread can start before the fail_loc is active.
      2. Add a stack_trap that sets fail_loc=0 and restarts any stopped MDT, so a failure here does not break the tests that follow.

            wc-triage WC Triage
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: