https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13563 Lustre <p>It would better to design a reclaim mechanism to free up some reserved inodes for newly creation or cache pages for latter I/O in case of cache saturation.<br/> We could use a kernel shrinker daemon that runs periodically:</p> <ul class="alternate" type="square"> <li>Unreserve inodes cached in MemFS.</li> <li>Commit the cache pages from MemFS into Lustre (assimilation phase) and unaccount all cached pages from the MemFS limits.</li> </ul> <p>The cache shrinker starts to work if the cache allocation has become larger than the upper watermark and it evicts files until the allocation is below a lower watermark.</p> LU-13563

WBC: Reclaim mechanism for cached inodes and pages under limits in MemFS

Technical task LU-10938 Minor Open Unresolved Qian Yingjin Qian Yingjin Fri, 15 May 2020 14:25:31 +0000 Mon, 13 Feb 2023 08:37:09 +0000 0 4 <p>Could we just hook into the existing kernel inode/slab/page shrinkers to manage this? One thing that is important to remember is that these shrinkers are essentially a "notification method" from the kernel about memory pressure, but we should still be free to add/modify the inodes/pages that are being flushed at one time to be more IO/RPC friendly (e.g. selecting contiguous pages to write to the OST, though I'm not sure what would be best for MDT aggregation).</p> <p>One thing that we have to worry about is delaying writeback to the MDT/OST for too long, as that can cause memory pressure to increase significantly, and we will have wasted tens of seconds not sending RPCs, which could have written GBs of dirty data during that time. I think as much as possible it makes sense to have a "write early, free late" kind of policy that we have for dirty file data so that we don't waste the bandwidth/IOPS just waiting until we are short of memory.</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38697" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38697</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13563" title="WBC: Reclaim mechanism for cached inodes and pages under limits in MemFS" class="issue-link" data-issue-key="LU-13563">LU-13563</a> wbc: lfs wbc unreserve command to reclaim inodes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: be6a95ca03a262d9e7ec3330a94be7b4d086b921</p> <p>"</p> <p>One thing that we have to worry about is delaying writeback to the MDT/OST for too long, as that can cause memory pressure to increase significantly, and we will have wasted tens of seconds not sending RPCs, which could have written GBs of dirty data during that time. I think as much as possible it makes sense to have a "write early, free late" kind of policy that we have for dirty file data so that we don't waste the bandwidth/IOPS just waiting until we are short of memory.</p> <p>"</p> <p>Can we tune the kernel writeback parameters to achieve this goal?</p> <p>Linux Writeback Settings</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Variable</th> <th class='confluenceTh'>Description</th> </tr> <tr> <td class='confluenceTd'>dirty_background_ratio</td> <td class='confluenceTd'>As a percentage of total memory, the number of pages at which the flusher threads begin writeback of dirty data.</td> </tr> <tr> <td class='confluenceTd'>dirty_expire_centisecs</td> <td class='confluenceTd'> In milliseconds, how old data must be to be written out the next time a flusher thread wakes to perform periodic writeback.</td> </tr> <tr> <td class='confluenceTd'>dirty_ratio </td> <td class='confluenceTd'>As a percentage of total memory, the number of pages a process generates before it begins writeback of dirty data.</td> </tr> <tr> <td class='confluenceTd'>dirty_writeback_centisecs</td> <td class='confluenceTd'>In milliseconds, how often a flusher thread should wake up to write data back out to disk.</td> </tr> </tbody></table> </div> <p> </p> <p>Moreover, for data IO pages, we can control the limit of cache pages in MemFS per file to allow data caching in MemFS. If exceed this threshold (i.e. max_pages_per_rpc: 16M? or only 1M to allow to cache much more small files), the client will assimilate the cache pages from MemFS into Lustre. After that, all data IO on this file is directed to Lustre OSTs via Lustre normal IO path.</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38739" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38739</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13563" title="WBC: Reclaim mechanism for cached inodes and pages under limits in MemFS" class="issue-link" data-issue-key="LU-13563">LU-13563</a> wbc: reclaim mechanism for inodes cached in MemFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6bc9b4310c573426e62166168e55c6cce972bbbb</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38749" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38749</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13563" title="WBC: Reclaim mechanism for cached inodes and pages under limits in MemFS" class="issue-link" data-issue-key="LU-13563">LU-13563</a> wbc: reclaim mechanism for pages cached in MemFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 66944d44e32abbb8a8332746cfa2fa5961267515</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38875" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38875</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13563" title="WBC: Reclaim mechanism for cached inodes and pages under limits in MemFS" class="issue-link" data-issue-key="LU-13563">LU-13563</a> mdt: ignore quota when creating slave stripe<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d497a600c487fd62401d776cea7d18644a74d4e2</p> Related Development Rank 1|i010gf: Rank (Obsolete) 9223372036854775807