[LU-15300] mirror resync can cause EIO to unrelated applications

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.15.3
• Labels: LTS15 failing_tests flr-improvement
• Severity: 3
• Epic Link: FLR tech debt review
• Assignee: Alex Zhuravlev
• Reporter: Alex Zhuravlev

Description

I noticed that sometimes sanity-flr/200 hits "checksum error", here are some findings.

first of all, checksum error is caused by incomplete preceding lfs mirror resync command (which doesn't return an error in some cases).

in turn, EIO lfs hits is caused by AS_EIO flag on the corresponded mapping.

AS_EIO is set because of ESTALE to OST_WRITE with incorrect layout version (client's version is smaller than one on OST).

so far I've traced all this to the race between two processes:

• lfs doing resync and changing layout generation
• another process (say, multiop) doing regular write

I will cite the logs in a subsequent comment.

Issue Links

• is duplicated by: LU-14966 sanity-flr test_200: FAIL: checksum error for mirror 2: lfs mirror: '/mnt/lustre/f200.sanity-flr' llapi_mirror_resync_many: Input/output error
• is related to: LU-18476 interop replay-single test_202: FAIL: layout gen changed: 2 -> 0
• is related to: LU-12656 sanity-flr test 200 fails with 'failed writing to *:*’
• is related to: LU-17070 sanity-flr test_200b: vvp_vmpage_error()) LBUG
• is related to: LU-15269 sanity-flr/200 to generate new tmp files each time
• is related to: LU-18416 Data corruption/miscompare observed during 48hr FOFB