-
Bug
-
Resolution: Fixed
-
Medium
-
None
-
None
-
FLR files during OSS failover
-
3
-
9223372036854775807
While the OSTs of a mirrored file are unavailable (an OSS failover, a dead OSS, also the MDT for the DoM component), reads of the file fail with EIO, either within milliseconds or after one RPC timeout. Reads of a file with one mirror wait for the targets and succeed. On a test cluster, a 120 s outage gave thousands of EIO on mirrored files and none on files with one mirror.
Reproducer
Two OSTs:
lfs setstripe -N -Eeof -o0 -N -Eeof -o1 f dd if=/dev/urandom of=f bs=1M count=4; lfs mirror resync f dd if=/dev/urandom of=f bs=4k count=1 seek=10 conv=notrunc # one mirror stale # stop both OSTs, wait until the client's OSC imports are no longer FULL, then cat f > /dev/null # Input/output error at once, not waiting for the OSTs
Cause
A read is non-delay. When the OSC import is not healthy it fails with -EAGAIN and cl_io_loop() restarts it on the next mirror. After four rounds over the mirrors lov_io_mirror_init() sets ci_tried_all_mirrors to stop the fast switching, but ci_ndelay stays set: the lock enqueue (LDLM_FL_NDELAY) and the BRW (rq_no_delay) fail at once with -EAGAIN, which cl_io_loop() turns into -EIO.
Fix
Once all mirrors were tried, the read waits for its targets like a read of a file with one mirror, until they are back or the client is evicted. It waits for the mirror it is on in that round only. Patch with a sanity-flr test: https://review.whamcloud.com/69949