Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21020

Read of a mirrored file fails with EIO during an OST failover

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • FLR files during OSS failover
    • 3
    • 9223372036854775807

      While the OSTs of a mirrored file are unavailable (an OSS failover, a dead OSS, also the MDT for the DoM component), reads of the file fail with EIO, either within milliseconds or after one RPC timeout. Reads of a file with one mirror wait for the targets and succeed. On a test cluster, a 120 s outage gave thousands of EIO on mirrored files and none on files with one mirror.

      Reproducer

      Two OSTs:

      lfs setstripe -N -Eeof -o0 -N -Eeof -o1 f
      dd if=/dev/urandom of=f bs=1M count=4; lfs mirror resync f
      dd if=/dev/urandom of=f bs=4k count=1 seek=10 conv=notrunc    # one mirror stale
      # stop both OSTs, wait until the client's OSC imports are no longer FULL, then
      cat f > /dev/null    # Input/output error at once, not waiting for the OSTs
      

      Cause

      A read is non-delay. When the OSC import is not healthy it fails with -EAGAIN and cl_io_loop() restarts it on the next mirror. After four rounds over the mirrors lov_io_mirror_init() sets ci_tried_all_mirrors to stop the fast switching, but ci_ndelay stays set: the lock enqueue (LDLM_FL_NDELAY) and the BRW (rq_no_delay) fail at once with -EAGAIN, which cl_io_loop() turns into -EIO.

      Fix

      Once all mirrors were tried, the read waits for its targets like a read of a file with one mirror, until they are back or the client is evicted. It waits for the mirror it is on in that round only. Patch with a sanity-flr test: https://review.whamcloud.com/69949

      Related: LU-12669, LU-17973

            markus.hilger Markus Hilger
            markus.hilger Markus Hilger
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: