https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6842 Lustre <p>The purpose is that the client can cache max_cached_mb at maximum on the client side but if the memory is in pressure it should consume less by hooking it up to cache shrinker.</p> LU-6842

Seeking the option to hook cl_page LRU up to kernel cache shrinker

Improvement Minor Resolved Fixed Zhenyu Xu Jinshan Xiong llnl Mon, 13 Jul 2015 17:23:12 +0000 Tue, 8 Oct 2019 08:04:22 +0000 Fri, 9 Oct 2015 20:39:43 +0000 Lustre 2.8.0 0 9 <p>Bobijam</p> <p>Could you please investigate what is possible here?</p> <p>Peter</p> <p>So this shrinker just reduce the ccc_lru_max, and of course shrink LRU if ccc_lru_left is not enough. I'm wondering when the ccc_lru_max should be restored gradually when the memory is not under such pressure?</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/15630" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15630</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6842" title="Seeking the option to hook cl_page LRU up to kernel cache shrinker " class="issue-link" data-issue-key="LU-6842"><del>LU-6842</del></a> clio: limit max cl_page LRU thru shrinker<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a7cf7b259c93d0f30f390b142878f137d7791ba4</p> <blockquote> <p>So this shrinker just reduce the ccc_lru_max, and of course shrink LRU if ccc_lru_left is not enough. I'm wondering when the ccc_lru_max should be restored gradually when the memory is not under such pressure?</p></blockquote> <p>Sorry for delay response.</p> <p>This is not about reducing ccc_lru_max, which we don't need to adjust in this case. Actually the meaning of ccc_lru_max should be 'to cache this many pages if memory is enough'. When the system memory is under pressure, Lustre shouldn't cache that much memory at all.</p> <p>We should register a cache shrinker on the OSC layer to get rid of some pages under memory pressure. Please take a look at osc_lru_reclaim(), but instead we are going to iterate each individual client_obd and destroy pages from them by the policies of last use, # of pages cached, etc.</p> <p>patch updated.</p> <p>Test results from Hyperion</p> <p>Cliff - Can you explain the tests results a bit more? What's memhog, for example? Is all the testing with the patch installed? If so, are there non-patched results somewhere to compare to?</p> <p>memhog is a test program under $LUSTRE/lustre/test. It just allocates a huge amount of memory so that we can investigate if Lustre does well under memory pressure.</p> <p>Added 2.7.56 results for comparison</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15630/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15630/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6842" title="Seeking the option to hook cl_page LRU up to kernel cache shrinker " class="issue-link" data-issue-key="LU-6842"><del>LU-6842</del></a> clio: add cl_page LRU shrinker<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 888a3141e72a25bef8daf822325b4295e5a0d5e8</p> <p>Landed for 2.8</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bobijam" class="user-hover" rel="bobijam">bobijam</a>,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> +#ifndef HAVE_SHRINKER_COUNT +<span class="code-keyword">static</span> <span class="code-object">int</span> osc_cache_shrink(SHRINKER_ARGS(sc, nr_to_scan, gfp_mask)) +{ + struct shrink_control scv = { + .nr_to_scan = shrink_param(sc, nr_to_scan), + .gfp_mask = shrink_param(sc, gfp_mask) + }; +#<span class="code-keyword">if</span> !defined(HAVE_SHRINKER_WANT_SHRINK_PTR) &amp;&amp; !defined(HAVE_SHRINK_CONTROL) + struct shrinker *shrinker = NULL; +#endif + + (void)osc_cache_shrink_scan(shrinker, &amp;scv); + + <span class="code-keyword">return</span> osc_cache_shrink_count(shrinker, &amp;scv); +} +#endif </pre> </div></div> <p>Is there any particular reason to return the value from osc_cache_shrink_count() instead of osc_cache_shrink_scan() itself?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> kswapd0-43 [003] .... 2885.831749: mm_shrink_slab_start: osc_cache_shrink+0x0/0x60 [osc] ffff8800cd67c1c0: objects to shrink 56 gfp_flags GFP_KERNEL pgs_scanned 222 lru_pgs 665761 cache items 645137 delta 430 total_scan 486 kswapd0-43 [003] d... 2885.831751: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x0 kswapd0-43 [003] d... 2885.831752: r_osc_cache_shrink_0: (shrink_slab+0x15c/0x340 &lt;- osc_cache_shrink) arg1=0x9d811 kswapd0-43 [003] d... 2885.832371: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x80 kswapd0-43 [003] d... 2885.832374: r_osc_cache_shrink_0: (shrink_slab+0x175/0x340 &lt;- osc_cache_shrink) arg1=0x9d791 kswapd0-43 [003] d... 2885.832377: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x0 kswapd0-43 [003] d... 2885.832378: r_osc_cache_shrink_0: (shrink_slab+0x15c/0x340 &lt;- osc_cache_shrink) arg1=0x9d791 kswapd0-43 [003] d... 2885.833002: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x80 kswapd0-43 [003] d... 2885.833004: r_osc_cache_shrink_0: (shrink_slab+0x175/0x340 &lt;- osc_cache_shrink) arg1=0x9d711 kswapd0-43 [003] d... 2885.833008: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x0 kswapd0-43 [003] d... 2885.833009: r_osc_cache_shrink_0: (shrink_slab+0x15c/0x340 &lt;- osc_cache_shrink) arg1=0x9d711 kswapd0-43 [003] d... 2885.833569: r_osc_cache_shrink_scan_0: (osc_cache_shrink+0x36/0x60 [osc] &lt;- osc_cache_shrink_scan) arg1=0x80 kswapd0-43 [003] d... 2885.833571: r_osc_cache_shrink_0: (shrink_slab+0x175/0x340 &lt;- osc_cache_shrink) arg1=0x9d691 kswapd0-43 [003] .... 2885.833573: mm_shrink_slab_end: osc_cache_shrink+0x0/0x60 [osc] ffff8800cd67c1c0: unused scan count 56 <span class="code-keyword">new</span> scan count 102 total_scan 46 last shrinker <span class="code-keyword">return</span> val 644753 </pre> </div></div> <p>It seems like in such a scenario, vmscan requested 3 times to scan 128 objects, osc_cache_shrink_scan() reported 3 times that 128 objects were freed. However, the shrinker returned not 0x80;0x80;0x80, but 0x9d791; 0x9d711; 0x9d691, reported as "last shrinker return val 644753".</p> <p>P.S. arg1 is $retval for osc_cache* retprobes.</p> Related LU-5841 Development Rank 1|hzxi2f: Rank (Obsolete) 9223372036854775807