Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20833

replay_barrier lets the server act on commits it then discards

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      replay_barrier is meant to simulate a server crash at the barrier. It does two things:

      • notransno stops the target from reporting last_committed to clients;
      • the device is switched to dm-flakey drop_writes (lctl readonly on ZFS).

      Neither stops the target from seeing its own commits. dm-flakey completes every dropped write as a success, so jbd2 commits normally and the commit callbacks run with err == 0. tgt_cb_last_committed() then raises obd_last_committed and exp_last_committed and calls ptlrpc_commit_replies() and tgt_cancel_slc_locks(). It checks neither OBDF_NO_TRANSNO nor anything else the barrier sets.

      So every server-side consumer of a commit acts on a transaction that failover then throws away:

      • COS/TXN locks saved in difficult replies are released (ptlrpc_commit_replies -> ptlrpc_handle_rs);
      • saved cross-MDT locks are cancelled (tgt_cancel_slc_locks);
      • sync-on-cancel (tgt_blocking_ast -> tgt_sync) returns success;
      • th_sync transactions report success, including FID sequence grants and chmod/chown with sync_permission=1, the default;
      • OSP sync (osp_sync_local_commit_cb) sends OST destroys and setattrs for MDT changes that failover then drops.

      A real crash cannot produce these histories. Take DNE commit-on-sharing:

      1. Op1 on MDT A parks its locks as LCK_TXN.
      2. Op2 on MDT B conflicts with them, which forces a commit on A.
      3. The commit "completes" into the dropped device, and op2 is granted.
      4. After failover, op1 is gone from A's disk, while op2 has run on B on the strength of op1's commit.

      A real crash gives one of two histories: op1 is durable before op2 runs, or op2 is never granted. Under the barrier, recovery replays a pair of operations that commit-on-sharing exists to keep apart, so a test can fail, or pass, on a history no crash produces.

      This has been known for single cases, each fixed or worked around on its own:

      • the pre-barrier mcreate in replay_barrier (b=24078), so no FID sequence is allocated after it;
      • LU-1573 drops the sync BRW reply under OBDF_NO_TRANSNO;
      • LU-3536 freezes the update-log batchid under the same flag;
      • replay-vbr turns COS and sync_permission off.

      LU-3140 recorded the general limitation: "Any sync operation will be a problem, because it will finish like non-sync". LU-14470 noted that DNE sync-on-cancel "can't be tested with replay_barrier()". Nothing in test-framework or the tests records either.

      Tests exposed today: replay-single 70b-70e and replay-dual 26 run DNE load across repeated barriers, so SLC acts on lost commits there. Some of their long-standing intermittent failures may be this (for example LU-14878, LU-15892, LU-18001, LU-10559); that is not established.

      Proposed fix

      1. A test-only target flag set by the barrier, e.g. OBDF_REPLAY_BARRIER with its own sysfs attribute. It must be separate from OBDF_NO_TRANSNO, which production failover umount also sets.
      2. With the flag set, tgt_cb_last_committed() returns early, as it already does for err != 0: last_committed does not advance, and saved COS/TXN and SLC locks stay held until the target stops. osp_sync_local_commit_cb() does the same.
      3. Synchronous waiters that bypass the callbacks (tgt_sync from tgt_blocking_ast, th_sync transactions, OST_SYNC/MDS_SYNC) must not return success under the flag: drop the reply as LU-1573 does, or block until target cleanup with -ESHUTDOWN.

      Expected test impact: recovery-small 156 relies on its O_SYNC write completing after the barrier and needs the write backgrounded across the fail; replay-single 115 and 137a/137c need a run; the DNE load tests keep their assertions, with brief stalls at each barrier.

      Risks: parked service threads hold ldlm locks, so umount must unwind them; many parked sync operations can use up MDT service threads until the fail. HARD failover (a real reboot) is unaffected.

            wc-triage WC Triage
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: