https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12970 Lustre <p>there are number of reports demonstrating a poor behaviour of mballoc on huge filesystems. in one report it was 688TB filesystem with 5.3M groups.<br/> mballoc tries to allocate large chunks of space, for small allocations it tries to preallocate and share large chunks. while this is good in terms of fragmentation and streaming IO allocation itself may need to scan many groups to find a good candidate.<br/> mballoc maintains internal in-memory structures (buddy cache) to speed up searching, but that cache is built from regular on-disk bitmaps, meaning IO. and if cache is cold, populating it may take a lot of time.</p> <p>there are few ideas how to improve that:</p> <ul> <li>skip more groups using less information when possible</li> <li>stop scanning if too many groups have been scanned (loaded) and use best found</li> <li>prefetch bitmaps (use lazy init thread? prefetch at scanning)</li> </ul> <p>another option for prefetching would be to skip non-initialized groups, but start an async read for the corresponding bitmap.<br/> also, when mballoc marks the blocks used (allocation has been just made) it could make sense to check/prefetch the subsequent group(s) which is likely a goal for subsequent allocation - while the caller are writting IO to just allocated blocks, the next group(s) will be prefetchted and ready to use.</p> LU-12970

improve mballoc for huge filesystems

Improvement Minor Open Unresolved WC Triage Alex Zhuravlev ldiskfs Thu, 14 Nov 2019 16:56:49 +0000 Wed, 7 Jun 2023 01:20:57 +0000 0 10 <p>I think that prefetching the block bitmaps in large chunks should be relatively easily implemented using the lazy_init thread. There is already patch <a href="https://review.whamcloud.com/32347" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32347</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10946" title="add an interface to load ldiskfs block bitmaps" class="issue-link" data-issue-key="LU-10946"><del>LU-10946</del></a> ldiskfs: add an interface to load ldiskfs block bitmaps</tt>" but I suspect that this is happening only after the mount, which is too late to be useful, and it is loading the bitmaps one-at-a-time, which causes a lot of extra overhead. Also, there was objection upstream to the extra data structure used to reference the bitmaps.</p> <p>Instead, the block bitmap prefetch should be done a whole flex_bg at a time (256 blocks), asynchronously during mount and the buddy and group info calculated in the end_io completion handler. It would make sense to keep the same sysfs interface to allow pinning the bitmaps as 32347 to maintain compatibility. </p> <p>Reducing size expectations for allocations during mount, and/or limiting scanning should also help. I think for small writes, we should avoid trying to do group preallocation until after the bitmaps have been loaded. That can be handled entirely inside the ldiskfs code and avoids the need to understand what is happening at the Lustre level.</p> <p>The bitmap scanning code can also advance the allocation hints itself until it finds some groups that have suitable free space, instead of waiting for an incoming write to do this. </p> <p>I cooked a new patch before to load block bitmaps async using workqueue:</p> <p><a href="http://gerrit.datadirectnet.jp:8082/#/c/2965/2/ldiskfs/kernel_patches/patches/rhel7.6/ext4-loadbitmaps.patch" class="external-link" target="_blank" rel="nofollow noopener">http://gerrit.datadirectnet.jp:8082/#/c/2965/2/ldiskfs/kernel_patches/patches/rhel7.6/ext4-loadbitmaps.patch</a></p> <p>And there was interface to control how much blocks could be prefetched each time, but havne't got some benchmark numbers for it yet.</p> <p> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/33874/33874_ext4-loadbitmaps.patch" title="ext4-loadbitmaps.patch attached to LU-12970">ext4-loadbitmaps.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>I've got a script to prepare a fragemnted filesystem using debugfs's setb and freeb commands which basically takes few seconds.<br/> so now I can reproduce this issue easily - I see one by one bitmap load of few hundred non-empty groups initiated by a single-block allocation.<br/> the next step is to add some instrumentation..</p> <p>Alex, I think that setting "RAID stripe size" (<tt>sbi-&gt;s_stripe</tt>) in the superblock may also be a contribute to the problem. For large RAID systems this is typically 512 blocks (2MB), up to 2048 blocks (8MB) or more in order to get allocations sized an aligned with the underlying RAID geometry. That in itself is good for large writes, but for small writes at mount time it can be problematic.</p> <p>Shilong, could you please post your patch to WC Gerrit so that it can be reviewed. Once the block bitmap is loaded, it makes sense to call <tt>mb_regenerate_buddy()</tt> to create the buddy bitmap and <tt>ext4_group_info</tt> as part of the <tt>ext4_end_bitmap_read()</tt> callback rather than waiting in <tt>ext4_wait_block_bitmap()</tt> for the bitmaps. That allows submitting IO in batches and letting it complete asyncrhonously (keep an atomic counter of how many blocks need to be processed and submit more IO when it gets large enough), rather than doing read then wait for all blocks to finish, read/wait, ...</p> <blockquote> <p>I've got a script to prepare a fragmented filesystem using debugfs's setb and freeb commands which basically takes few seconds.</p></blockquote> <p>Alex, it would be very useful to submit this upstream to e2fsprogs, since testing fragmented filesystems is always a problem.<br/> It also makes sense for you to see if Shilong's current patch helps your test case, and then we can work on optimizing it further.</p> <p>sure, will try to make the script useful for the outter world.</p> <p><a href="https://review.whamcloud.com/#/c/36793/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/36793/</a> - this patch limits scanning for a good goup and adds basic prefetching.<br/> currently it's more like an RFC, though I tested it manually</p> <p>Alex, is this complementary with the <a href="https://jira.whamcloud.com/browse/LU-12103" title="Improve block allocation for large partitions" class="issue-link" data-issue-key="LU-12103"><del>LU-12103</del></a> patch that is already landed?</p> <p>I think it's a bit different approach. overall fullness doesn't mean we can't find good chunks, IMO.<br/> say, few files have been written very dense so that 1/2 of groups are full, but another 1/2 is nearly empty.<br/> why should we change the algorithm?</p> <p>While it is possible to have the 1/2 full and 1/2 empty groups case you propose, I don't think that this is a likely condition. Even so, in this case, wouldn't the allocator just find the first empty group and allocate linearly from there?</p> <p>hmm, why you think this is not likely? few growing files would fill the filesystem group by group.<br/> "just find" - this is exactly the issue. the allocator is supposed to be generic enough to work with small and big files, right?<br/> thus we want to keep some locality, if file A has last extent in the group N, then we should try to write next extent in the same N or nearby, not just any empty group?<br/> and then searching for the group is what is happening in DDN -923, but the groups weren't considered "best" and that got worse due to cold cache.<br/> so that approach I'm trying is to limit coverage of searching.<br/> I think that coverage can be expressed in number of groups to search in and/or number of uninitialized groups causing IO.<br/> on the first try we can search for exactly requested chunk in N groups, if failed relax requirement and search for best in N*m groups, then just anything..</p> <p>partly for curiosity attached old SATA 7200 500GB drive to my testing box..:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@rz /]# time cat /proc/fs/ext4/sda/mb_groups &gt;/dev/<span class="code-keyword">null</span> real 0m24.081s user 0m0.000s sys 0m0.274s </pre> </div></div> <p>this is 3726 groups, initialized by mke2fs so all to read during that cat.</p> <p>with 32-groups-at-once prefetching, the same cat:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> real 0m14.150s user 0m0.000s sys 0m0.309s </pre> </div></div> <p>with 64-groups-at-once prefetching:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> real 0m13.200s user 0m0.000s sys 0m0.277s </pre> </div></div> <p>but this is a single spindle, for any regular site that would be multiple spindles I guess and a larger prefetch window would help more.</p> <p>given in all the case we do forward scan, I think it would be relatively simple to add few lists of groups to be scanned at each criterion.<br/> each time a group gets/loses free blocks we would move it from one to another list at cost of few CPU cycles.<br/> I'm not sure what are crossing points for each list yet (in terms of free blocks/fragments/etc), but the inital implementation<br/> could start with empty/non-empty lists.</p> <p>I have thought in the past about something similar to what you describe. However, it is difficult to know in advance what the size requirements are.</p> <p>One though was whether it makes sense to have a higher-level buddy bitmap for groups that is generated at the default preallocation unit size (based on the <tt>s_mb_large_req</tt> size) that allows quickly finding groups that have available 8MB or 16MB chunks, up to the maximum possible allocation size (probably 64MB is enough). At 8-64MB chunks this would mean 15MB of bitmap for a 512TiB filesystem (could use <tt>kvmalloc()</tt>). This would be essentially a filesystem-wide replacement for the <tt>bb_counters</tt> array that is tracked on a per-group basis, so would likely reduce overall memory usage, and would essentially replace "group scanning" with "bitmap scanning". It could be optimized to save the first set bit to avoid repeatedly scanning the blocks of beginning of the filesystem, assuming they would be preferentially allocated.</p> <p>This could also be implemented as an array of linked lists (at power-of-two granularity up to 64MB), with groups being put in the list with their largest aligned free chunk (separate lists for unaligned chunks?). Allocations would first walk the list for the smallest chunk that they need, then move up to lists with progressively larger chunks if no groups are available at the smaller size. Once the allocation is done, the group may be demote to to a lower list if the allocation results in a smaller chunk being available. To add a <tt>list_head</tt> to each of 4M groups in a 512TiB filesystem would consume 64MB of memory, but it would be split across all of the <tt>ext4_group_info</tt> allocations.</p> <p>Note that using a bigalloc size of e.g. 32KB would reduce the number of groups by a factor of 8 (e.g. 4M -&gt; 512K) so we should also consider fixing the issues with bigalloc so that it is usable.</p> <p>Link to backport of upstream mballoc patches in <a href="https://jira.whamcloud.com/browse/LU-14438" title="backport ldiskfs mballoc patches" class="issue-link" data-issue-key="LU-14438">LU-14438</a>, which may be enough to resolve this issue.</p> Related LU-14438 LU-12103 LU-10946 LU-8365 LU-15319 LU-12976 LU-16155 LU-12988 LU-16691 Development Rank 1|i00phj: Rank (Obsolete) 9223372036854775807