https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14639 Lustre <p>There is an configure option to enable/disable lru-resize (--enable-lru-resize/<br class="atl-forced-newline" />--disable-lru-resize), but that confuses when lru_size set 0.<br/> In both cases, it's able to set 0 to lru_size, but they make different behaviors.</p> <p>And currently, if lru-resize disabled(--disable-lru-resize) when lustre builds, it won't be able to re-enable lru-resize even lru_size set 0.</p> <p>./configure --enable-lru-resize</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sky06 ~]# mount -t lustre 10.0.11.110@o2ib10:/ai7990 /ai7990 mount -t lustre 10.0.11.110@o2ib10:/ai7990 /ai7990 mount.lustre: according to /etc/mtab 10.0.11.110@o2ib10:/ai7990 is already mounted on /ai7990 [root@sky06 ~]# lctl get_param ldlm.*.*.lru_size ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=8800 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8f5408893800.lru_size=1 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-OST0000-osc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-OST0001-osc-ffff8f5408893800.lru_size=0 [root@sky06 ~]# lctl set_param ldlm.*.*.lru_size=0 ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=0 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-OST0000-osc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-OST0001-osc-ffff8f5408893800.lru_size=0 [root@sky06 ~]# time find /ai7990/testdir &gt; /dev/null 2&gt;&amp;1 real 0m15.183s user 0m0.367s sys 0m6.587s [root@sky06 ~]# lctl get_param ldlm.*.*.lru_size ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=0 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8f5408893800.lru_size=30502 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8f5408893800.lru_size=0 ldlm.namespaces.ai7990-OST0000-osc-ffff8f5408893800.lru_size=2499 ldlm.namespaces.ai7990-OST0001-osc-ffff8f5408893800.lru_size=2501 </pre> </div></div> <p>./configure --disable-lru-resize</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sky06 ~]# mount -t lustre 10.0.11.110@o2ib10:/ai7990 /ai7990 [root@sky06 ~]# lctl get_param ldlm.*.*.lru_size ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=8800 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8ef7ae5ef000.lru_size=8800 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8ef7ae5ef000.lru_size=8800 ldlm.namespaces.ai7990-OST0000-osc-ffff8ef7ae5ef000.lru_size=8800 ldlm.namespaces.ai7990-OST0001-osc-ffff8ef7ae5ef000.lru_size=8800 [root@sky06 ~]# lctl set_param ldlm.*.*.lru_size=0 ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=0 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-OST0000-osc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-OST0001-osc-ffff8ef7ae5ef000.lru_size=0 [root@sky06 ~]# time find /ai7990/testdir &gt; /dev/null 2&gt;&amp;1 real 0m26.491s user 0m0.358s sys 0m11.809s [root@sky06 ~]# lctl get_param ldlm.*.*.lru_size ldlm.namespaces.MGC10.0.11.110@o2ib10.lru_size=0 ldlm.namespaces.ai7990-MDT0000-mdc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-MDT0001-mdc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-OST0000-osc-ffff8ef7ae5ef000.lru_size=0 ldlm.namespaces.ai7990-OST0001-osc-ffff8ef7ae5ef000.lru_size=0 </pre> </div></div> master(commit:gdfe87b0) LU-14639

confusion of lru_size=0 if lru-resize disabled

Bug Minor Open Unresolved WC Triage Shuichi Ihara Sat, 24 Apr 2021 00:19:55 +0000 Tue, 30 May 2023 17:59:54 +0000 Lustre 2.15.0 0 8 <p>What is expected to be the right behavior here? <tt>&#45;&#45;disable&#45;lru&#45;resize</tt> removed the LRU resize code completely, so only fixed-size LRU is possible. Then setting <tt>lru_size=0</tt> results in no locks being cached on the clients. </p> <p>Why even build with <tt>&#45;&#45;disable&#45;lru&#45;resize</tt> these days, instead of just setting "<tt>lctl set_param &#45;P ldlm.namespaces.&lt;fsname&gt;&#42;.lru_size=500</tt>" or similar? </p> <p>The only option I see is to change <tt>&#45;&#45;disable&#45;lru&#45;resize</tt> to <b>not</b> actually disable the LRU resize code, and instead have it just set a fixed LRU size by default to prevent users from shooting themselves in the foot because they are using old instructions when building clients. I don't <em>think</em> this is documented anywhere, but if it is then it should be removed. </p> <blockquote><p>What is expected to be the right behavior here? <tt>--disable-lru-resize</tt> removed the LRU resize code completely, so only fixed-size LRU is possible. Then setting <tt>lru_size=0</tt> results in no locks being cached on the clients.</p></blockquote> <p>The problem is that <tt>lru_size=0</tt> is configurable regardless client was built with --disable-lru-resize or --enable-lru-resize, but people can't make judge which is which after changed <tt>lru_size=0</tt>. Even non zero value setting, it's hard to confirm. When client umounts and mounts lustre again, it can confirm by default value (zero or non-zero lru_size).<br/> There are two totally different behaviors, but controlling in same parameter and value.</p> <p>If client was built with <tt>--disable-lru-resize</tt>, setting <tt>lru_size=0</tt> shouldn't be acceptable?, but other value needs to be defined to give "no locks cache" meaning? e.g. <tt>lru_size=false</tt></p> <blockquote><p>Why even build with <tt>--disable-lru-resize</tt> these days, instead of just setting "<tt>lctl set_param -P ldlm.namespaces.&lt;fsname&gt;*.lru_size=500</tt>" or similar?</p></blockquote> <p>This is fine, but it still needs per client setting. client's UUID changes all time doesn't it? e.g. it would limit lru_size only for login or data mover nodes, etc.</p> <p>There was a discussion about having per-client tunables linked to nodemap in <a href="https://jira.whamcloud.com/browse/LU-11077" title="Client-specific tunable parameter configuration" class="issue-link" data-issue-key="LU-11077">LU-11077</a>, or possibly a client-local <tt>/etc/lustre/&lt;fsname&gt;-client.params</tt> file to set parameters at mount time. However, that has not been implemented.</p> <p>Shuichi, is the main goal of using <tt>&#45;&#45;disable&#45;lru&#45;resize</tt> to have a different/static LRU size on a small number of clients (e.g. login node or data mover), or is used for all clients in a cluster? Or is there some other issue with LRU resize that means it should be disabled entirely from the code (e.g. jitter on compute nodes, or other reasons to disable the code completely? Is this option widely used for all client builds, or only in specific cases?</p> <p>I'm wondering if the meaning of the <tt>&#45;&#45;disable&#45;lru&#45;resize</tt> option should be changed from <b>removing</b> the LRU resize code to just changing it to have a constant <tt>lru_size</tt> value? Is there really a time when "<tt>lru_size=0</tt>" should mean "cache zero locks" (which would be terrible for performance, as you see here)? If a client should minimize lock cache size, I can't imagine that <tt>lru_size=5</tt> or similar would cause many issues, and would at least still allow a few files to re-use locks on the client...</p> <p>In many cases, reason of '--disable-lru-resize' would have a limit of lock counts in cache per client on entire cluster. <br/> And people cleanup caches after job finishes. (e.g. integrated running 'lctl set_param ldlm.namespaces.*.lru_size=clear' command in job scheduler as a post script)</p> <blockquote><p>I'm wondering if the meaning of the --disable-lru-resize option should be changed from removing the LRU resize code to just changing it to have a constant lru_size value?</p></blockquote> <p>Indeed, still enabling LRU, but fixed value by default make sense.<br/> It's still possible to change 0 or even more lower/higher value if it needs. And we can also control LRU speed by lru_max_age parameter.</p> <p>It would be nice to drop the configure option entirely - I don't think changing defaults as a build time option is very good practice, I think it should be done with at runtime with settings unless there's a reason that doesn't work.  It's weird to be able to change something with both build configuration <b>and</b> runtime options.  I think it would be nice to get away from the idea of changing behavior with <em>build</em> time flags if that behavior can also be adjusted at runtime.</p> <p>I'm guessing though that since we have customers who are using this build option, it would be easier to just change it to set a default value, right?</p> <p>Actually, I was just thinking about this, and:</p> <p>I don't think changing the build flag to set a default is a very good idea.  A good default value seems very hard to pick.  Like, what is a good default value for lru_size?  What represents a good compromise between memory usage and performance?  Etc.  The 'correct' value depends on whether it is an MDC or an OSC connection, possibly changing if DOM is in use, and on client and server memory size, etc.  We have lru-resize specifically because the correct value is hard to choose - it is very system specific - so instead we choose the value dynamically.</p> <p>So I think instead we should encourage people who want to set a specific lru_size to 'do the right thing' by removing the build option.  They will then set the lru_size value they want in their configuration.  This is what the customers using the build option are doing anyway - they built with disable-lru-resize out of caution, but they are all manually setting specific lru_size values.  So the build option is never used by itself anyway.</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51165" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51165</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14639" title="confusion of lru_size=0 if lru-resize disabled" class="issue-link" data-issue-key="LU-14639">LU-14639</a> build: Remove disable-lru-resize config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f457d586f8b01fd1f69e4d761f216535bacb4ca</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51166" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51166</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14639" title="confusion of lru_size=0 if lru-resize disabled" class="issue-link" data-issue-key="LU-14639">LU-14639</a> tests: remove disable-lru-resize check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9e8e99f4f06bdb8bf5610ee7060bd6925c5e8c0a</p> Related LU-11077 Development Rank 1|i01t13: Rank (Obsolete) 9223372036854775807 Severity