https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16611 Lustre <p><tt>read_cache_enable</tt> is disabled by default if <tt>queue/rotational=0</tt></p> <p>I wanted to raise awareness that this might not be the best default value. I recently noticed that, at least on recent hardware (eg. AMD EPYC Milan) with SAS-based SSDs (not NVMe), it is more efficient to keep <tt>read_cache_enable</tt> enabled to improve the read performance with many fio benchmarks I ran (sequential I/Os and random I/Os), at least the ones that could benefit from OSS caching (I was careful to not benchmark the client cache...).</p> <p>This is at least my experience with SAS SSDs, but I understand it might be better to keep the OSS read cache disabled with NVMe drives when accessed directly.</p> osd-ldiskfs LU-16611

default read_cache_enable=0 not always the best choice when rotational=0

Improvement Minor Open Unresolved WC Triage Stephane Thiell Thu, 2 Mar 2023 03:15:34 +0000 Wed, 8 Mar 2023 05:36:45 +0000 Lustre 2.12.9 Lustre 2.15.2 0 3 <p>Stephane, I believe that this parameter is careful not to override any value set by the config, so it should be possible to change it in the normal manner (ie "<tt>lctl set_param -P</tt>". </p> <p>Also, since this only affects the in-memory cache, there is no problem if it is not set in the "ideal" manner for a second or two after mount while the OSS is applying the config parameters from the MGS. </p> <p>Since we have very little information available to the filesystem (it was a challenge even getting "<tt>rotational</tt>" tone set consistently for the storage), I don't think it is practical for Lustre to auto-tune this based on the specific storage type.</p> <p>Would this be best handled by describing the various options in the Lustre Operations Manual?</p> <p>Andreas, thanks for your reply, that makes sense. With servers' memory performance increasing and probably the Linux page cache's performance also improving (not sure of that), it might be worth adding a note in the manual.</p> <p>Today the Lustre manual says:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>By default, read cache is enabled (read_cache_enable=1) for HDD OSDs and automatically disabled for flash OSDs (nonrotational=1).  </pre> </div></div> <p>Perhaps adding something like that would be a good general guidance:</p> <p>However, depending on the hardware (performance of flash storage, type of server), it might be still beneficial to enable read_cache_enable with flash OSD. This can be done permanently with lctl set_param -P ....</p> <p>Patches welcome <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> You can also use the LU ticket to submit a patch to the manual, no need for a separate LUDOC ticket. </p> Development Rank 1|i03fhj: Rank (Obsolete) 9223372036854775807