[LU-14541] Memory reclaim caused a stale data read

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.0, Lustre 2.15.3
• Severity: 3
• Assignee: Patrick Farrell
• Reporter: Alexey Lyashkov

Description

1. Cray don't have a full logs described this problem, but big picture looks clean.
   Client node start a memory reclaim and enter to the ll_releasepage, where seen page is not a busy and have 3 vmpage references. It caused a cl_page_delete call which remove page from own page tree and move to the CPS_FREEDING state. It's fine for the kernels < 2.6.37.
   But 2.6.37 introduce a different way to page free, it is ->freepage callback.
   >>
   commit 6072d13c429373c5d63b69dadbbef40a9b035552
   Author: Linus Torvalds <[e-mail]>
   Date: Wed Dec 1 13:35:19 2010 -0500

Call the filesystem back whenever a page is removed from the page cache
>>
It introduced because remove_mapping() can prohibit to kill page from page cache due page refcount != 2, or PageDirty reasons. As page in CPS_FREEDING state, cl_page_own is failed to own a page in the blocking ast an code expect some else will free page, but none do it. OOPS. Stale page with uptodate flag set in the page cache - where it can read du fast read code path.
Some existent logs.
>>>
00000008:00100000:10.0:1615300198.692889:0:4147:0:(osc_cache.c:3288:osc_page_gang_lookup()) vvp-page@ffff8800310524e0(1:1) vm@ffffea000119bdd0 10000000000002c 4:0 0 82094 lru
bad
00000008:00100000:10.0:1615300198.692873:0:4147:0:(osc_cache.c:3279:osc_page_gang_lookup()) vvp-page@ffff8800310520e0(1:1) vm@ffffea000119be08 10000000000002c 3:0 0 82095 lru
good
>>>
Other logs show it's race between lock cancel (osc_gang_lookup) and kswapd.

so one more vmpage reference highly likely caused fail.
based from crash dump in second after it. Page have a two references.
so likely we have a race with page access.

Issue Links

• is related to: LU-16156 stale read during IOR test due LU-14541
• is related to: LU-12587 DIO fallback to Buffer IO unexpectedly
• is related to: LU-15815 fast_read/stale data/reclaim workround causes SIGBUS
• is related to: LU-16160 take ldlm lock when queue sync pages
• is related to: LU-15819 Executables run from Lustre may receive spurious SIGBUS signals
• is related to: LU-8633 SIGBUS under memory pressure with fast_read enabled

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50599/
Subject: Revert "LU-14541 llite: Check vmpage in releasepage"
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 72b5be5ccc1c58ae6edc968fa9106d53578aeccb

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50598/
Subject: LU-14541 llite: Check for page deletion after fault
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 4a134425bd9d03f5e40e489fd7e9acf4788e9da1

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49654/
Subject: Revert "LU-14541 llite: Check vmpage in releasepage"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e3cfb688ed7116a57b2c7f89a3e4f28291a0b69f

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49653/
Subject: LU-14541 llite: Check for page deletion after fault
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b3d2114e538cf95a7e036f8313e9095fe821da79

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50599
Subject: LU-14541 revert: "llite: Check vmpage in releasepage"
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 0926e0b3b61c8180024df55223f654080f789a6d

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50598
Subject: LU-14541 llite: Check for page deletion after fault
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 9cf02b117d1e51d8ad367bae88f8fda4c0a95f49

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49654
Subject: Revert "LU-14541 llite: Check vmpage in releasepage"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 95fb01cb020295531839d8c758320262db6b7bef