https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13533 Lustre <p>lazyinit gets more than 24H on typical OST installation and produces writes inside read.  This influences to benchmark tests that are usually executed just after cluster installation is complete. </p> <p>Testing shows, disabling that feature adds ~30 sec to formating OST drive so that is not a noticeable time during install.</p> LU-13533

Disable lazy_itable_init

Improvement Major Resolved Fixed Artem Blagodarenko Artem Blagodarenko patch Thu, 7 May 2020 17:32:34 +0000 Tue, 15 Feb 2022 15:42:08 +0000 Tue, 1 Sep 2020 04:57:15 +0000 Lustre 2.14.0 0 4 <p>Artem Blagodarenko (c17828@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/38534" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38534</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13533" title="Disable lazy_itable_init" class="issue-link" data-issue-key="LU-13533"><del>LU-13533</del></a> utils: ext4lazyinit should be disabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 35524f6c2c2fea4f20d8fda4ddfdf2f94dfd1253</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38534/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38534/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13533" title="Disable lazy_itable_init" class="issue-link" data-issue-key="LU-13533"><del>LU-13533</del></a> utils: ext4lazyinit should be disabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 701cc249594eae08f6e762eff74183e768c2cee3</p> <p>Landed for 2.14</p> <p>Artem, did you ever look into why the <tt>lazy_itable_init</tt> was taking so long in the kernel? Since <tt>mke2fs</tt> is also calling the same underlying <tt>blkdev_issue_zeroout()}] code (via {{fallocate(FALLOC_FL_ZERO_RANGE)</tt> on the block device) as the kernel (via <tt>ext4_issue_zeroout-&gt;sb_issue_zeroout()</tt>), it must be more of a scheduling issue from the <tt>ext4_lazyinit_thread</tt>.</p> <p>It looks like the default is to schedule the next group to start 10x the time it took to zero out the current group (default <tt>s_li_wait_mult = EXT4_DEF_LI_WAIT_MULT = 10</tt>), so at best it could use 10% of the available disk bandwidth. It is possible to use "<tt>-o lazy_itable_init=N</tt>" to tune this delay, although it looks like setting <tt>s_li_wait_mult=0</tt> or if the writes complete in less than 1 jiffie (maybe with flash) will cause all of the writes to be synchronous:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> timeout = jiffies; ret = ext4_init_inode_table(sb, group, elr-&gt;lr_timeout ? 0 : 1); <span class="code-keyword">if</span> (elr-&gt;lr_timeout == 0) { timeout = (jiffies - timeout) * elr-&gt;lr_sbi-&gt;s_li_wait_mult; elr-&gt;lr_timeout = timeout; } </pre> </div></div> <p>At a minimum, the above code should set <tt>elr-&gt;lr_timeout = timeout ?: 1</tt> so that it doesn't do all writes synchronously for fast storage.</p> <p>A better solution would be to schedule the inode table zeroing as fast as possible after mounting, and only throttle it down when the filesystem is "busy" with user requests. This could be done by using "trylock" variants for <tt>alloc_sem</tt>, etc. to detect if there is contention, or by having a simple percpu counter in some of the main incoming codepaths (e.g. lookup, read, write) and checking if this is changing between calls to detect if there is userspace activity. I'd thought maybe using the superblock <tt>s_kbytes_written</tt> counter for this, but it is only updated at unmount time, and from the underlying block device, so that would be affected by the zeroout itself.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> do you have plans to re-enable lazy_itable_init for Lustre?</p> <p>I've been thinking bout using the same mechanism for doing background TRIM operations, instead of the current "<tt>-o discard</tt>" code, so I would like to know what is wrong with the current code. </p> <p>While it might only add a few second to an OST it means our MDT's are going to take over 2 hrs EACH to format. We have like 8..... makes formatting MDT's a multi-day affair. </p> <p>Can we re-look at this?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=haaknci" class="user-hover" rel="haaknci">haaknci</a>, you can still use "<tt>--mkfsoptions='-E lazy_itable_init'</tt>" when formatting your MDTs. This is just a change to the default options because we had a number of complaints that performance after formatting the filesystem was slow. </p> <p>Not on 2.14.0 you can't. It has a bug. I've cherry picked the 2.15 fix. </p> Related LU-14767 LU-8144 Development Rank 1|i00zrz: Rank (Obsolete) 9223372036854775807