Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21289

DNE: live update llog cancel fails a cross-MDT operation in progress with EIO

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      lctl --device <MDT> llog_cancel on a running MDT can make a cross-MDT operation that is in progress fail with -EIO. The failure can possibly leave the operation partly done. See "Not checked" below.

      Trigger

      The ioctl goes mdt_iocontrol() -> mdd_iocontrol() -> lod_iocontrol() -> lod_llog_cancel(). LU-16159 added this path. lod_sub_cancel_llog() retains the update catalog of each MDT and installs a new, empty catalog.

      A distribute transaction declares its update record writes in top_trans_start() -> prepare_multiple_node_trans() -> declare_updates_write(). The declare reserves the next plain llog in the current catalog. It writes the records later, in top_trans_stop() -> sub_updates_write(). If the cancel replaces the catalog between the declare and the write, the write goes to the new catalog. The new catalog has no declared plain llog, so the write fails.

      Reproduction

      Master 2.17.62, ldiskfs, 3 MDTs on one VM. The VM had change 71205 applied. That change does not touch this path.

      1. Add a temporary 10 second CFS_FAIL_TIMEOUT in top_trans_start() after prepare_multiple_node_trans().
      2. Create dir1 on MDT0001 and dir2 on MDT0000. Create a file in dir1.
      3. Set the delay on the MDS, then run ln dir1/file dir2/file on the client.
      4. During the delay, run lctl --device lustre-MDT0000 llog_cancel --catalog=update_log --log_idx=1 on the MDS.

      The ln fails with EIO. The MDS console shows:

      Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0]
      Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x280000402:0x1:0x0]
      Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x240000402:0x1:0x0]
      LustreError: 12962:0:(update_trans.c:799:top_trans_start()) cfs_fail_timeout id 7ff awake
      LustreError: 12962:0:(llog_cat_server.c:407:llog_cat_current_log()) lustre-MDT0000-osd: next log does not exist, catalog [0x200000400:0x4:0x0] rc=-5
      LustreError: 12962:0:(update_trans.c:986:top_trans_stop()) lustre-MDT0000-osd: write updates failed: rc = -5
      

      The delay only makes the race easy to hit. Without it, the window is the operation body between top_trans_start() and top_trans_stop().

      Not checked

      • The state of the link after the failure: the name entry on MDT0000 and the nlink on MDT0001.
      • osd-zfs.

      Possible fixes

      • Make the cancel wait until no declared update record write is open. This adds a lock to each distribute transaction.
      • Refuse the update llog cancel on a running MDT. Allow it only in the recovery thread, which cancels the update llogs on recovery abort.

            wc-triage WC Triage
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: