https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8365 Lustre <p>Provide a mechanism to reset the ldiskfs extents allocation position to near the beginning of a drive</p> LU-8365

Fix mballoc stream allocator to better use free space at start of drive

Bug Major Open Unresolved Yang Sheng Lokesh Nagappa Jaliminche ldiskfs Mon, 4 Jul 2016 14:34:51 +0000 Tue, 10 Aug 2021 19:20:49 +0000 Lustre 2.8.0 0 14 <p>lokesh.jaliminche (lokesh.jaliminche@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/21142" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21142</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: procfs entries for mballoc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7218c37a694df7b0f057a6078dad24a9166300e7</p> <p>This patch exposes that mballoc is not doing as good a job in group selection for empty HDDs as it might. Biasing allocations to the start of the disk can improve performance, but only if the start of the disk has free space.</p> <p>Some possibilities to try that may actually fix mballoc, in order of increasing difficulty:<br/> 1) when freeing blocks (in the group descriptor update) if the group changes to less than X allocated blocks, and the group is below <tt>mb_last_group</tt> then reset <tt>mb_last_group</tt> to that group. In cases where a filesystem is being filled and emptied (e.g. benchmarks) this would automatically produce optimal results, without the need for this tunable at all. It would also work for all users. The threshold X should be larger than the count of bitmaps and inode table in a normal group. <br/> 2) As an added heuristic to the above, only go back if there are M consecutive groups that meet this criteria, so that we don't keep seeking back to one group that has only 2k allocated blocks and then scanning to the end of the used groups again. The number of groups (M) could be a tunable. <br/> 3) As an added heuristic to the above, save the old <tt>mb_last_group</tt> and return there if the current and next few groups are "full" (if <tt>(mb_last_group &gt; current group)</tt>), otherwise scan forward as usual. That avoids scanning a lot of potentially useless groups that have recently been scanned in order to get to where <tt>mb_last_group</tt> was previously. <br/> 4) as above, but using free chunk size instead of just free block count (e.g. use maximum free chunk size in the buddy bitmap). This could add in a few level(s) of free chunks, maybe above some threshold like 8MB with a simple shift+add <tt>hweight()</tt> loop while scanning the buddy bitmap.<br/> 5) instead of using <tt>mb_last_group</tt> at all, keep a "sorted" list or a tree of <tt>struct ext4_group_info</tt> that track free space in groups. The sorting comparison should prefer groups at the start of the disk in some way (eg. <tt>free_blocks - group_number/128</tt>). Populate this list/tree during mount-time group descriptor scan and keep it "sorted" as blocks are allocated and freed. Sorting can be lazy to avoid lots of rebalancing. Use the list/tree to find a good new target group if the current and next group are "full". This will also improve performance when the filesystem becomes nearly full, to avoid lots of scanning for groups with free blocks. <br/> 6) as above, but use separate list/trees based on fullness. No need to keep groups in sorted lists/trees if they are totally empty. No need to manage groups that are nearly full until enough blocks are freed to make them interesting allocation targets. It may be that this is _more_complex than a single list/tree, but it depends on how much the ongoing tree balancing costs. If the cost is high, and groups usually change from "mostly empty" to "mostly full" then having a "full" list to keep groups until they become "nearly empty" again would be useful.</p> <p>Thanks for the details, working on it.</p> <p>Hi Yang Sheng,<br/> would you be able to work on a patch for this issue? I think the first 3-4 steps in the proposed solution shouldn't be too hard to implement. It looks like <tt>mb_set_largest_free_order()</tt> or in its callers might be the right place to see if <tt>s_mb_last_group</tt> should be updated? We already track <tt>bb_free</tt> and <tt>bb_largest_free_order</tt> for each group, so they can be used to decide if we should reset <tt>s_mb_last_group</tt> or not. It would be good to save the old <tt>s_mb_last_group</tt> to return if scanning in <tt>ldiskfs_mb_regular_allocator()</tt> doesn't find any good groups quickly.</p> <p>Please feel free to ask if you have questions. I'd like to have something to look at late next week, if possible. We need to run some benchmarks on real hardware to ensure this is doing the right thing. It would be OK to include the patch <a href="https://review.whamcloud.com/21142" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21142</a> for testing/debugging, but I don't consider that a real fix for this issue.</p> <p>In a semi-related area, I also noticed in the current <tt>ext4-prealloc.patch</tt> while looking at this issue that there is a bug in the code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* don't use group allocation <span class="code-keyword">for</span> large files */</span> size = max(size, isize); + <span class="code-keyword">if</span> ((ac-&gt;ac_o_ex.fe_len &gt;= sbi-&gt;s_mb_small_req) || + (size &gt;= sbi-&gt;s_mb_large_req)) { ac-&gt;ac_flags |= EXT4_MB_STREAM_ALLOC; <span class="code-keyword">return</span>; } + /* + * request is so large that we don't care about + * streaming - it overweights any possible seek + */ + <span class="code-keyword">if</span> (ac-&gt;ac_o_ex.fe_len &gt;= sbi-&gt;s_mb_large_req) + <span class="code-keyword">return</span>; </pre> </div></div> <p>It looks like we can never get to the second condition because <tt>fe_len &gt; s_mb_small_req</tt> will always be true first. This has been true all the way back to the original version of this patch (commit <tt><a href="https://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=d8d8fd9192a54c7b8caef8cca9b7a1eb5e5e3298" class="external-link" target="_blank" rel="nofollow noopener">d8d8fd9192a5</a></tt>. It seems like the <tt>s_mb_large_req</tt> check should be moved before <tt>EXT4_MB_STREAM_ALLOC</tt> is set, so that it allows large allocations to behave differently?</p> <p>Hi, Alex,</p> <p>Looks like the 'stream allocation' has been changed since upstream patch(4ba74d00a2025). Could you please review it whether correct for original purpose. Other question is why we need s_mb_small_req? As i understand, 'stream allocation' would be used while request size less than s_mb_large_req. Then what purpose is s_mb_small_req? Could you give a point for that please?</p> <p>Thanks,<br/> YangSheng</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33195" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33195</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: try to alloc block toward lower sectors<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7e6352641724a26685247a21be488e43401437eb</p> <p>Hi, Alex, </p> <p>Could you please give a advice for this patch? <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31104/31104_0001-ext4-Fix-bugs-in-mballoc-s-stream-allocation-mode.patch" title="0001-ext4-Fix-bugs-in-mballoc-s-stream-allocation-mode.patch attached to LU-8365">0001-ext4-Fix-bugs-in-mballoc-s-stream-allocation-mode.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Thanks,<br/> YangSheng</p> <p>I'm not sure why you attached the patch here? That is what gerrit is for. </p> <p>Hi, Andreas,</p> <p>This is the patch has already landed to upstream. I just want to get some input from Alex whether it is correct for stream allocation. Since it changes logic of stream allocation.</p> <p>Thanks,<br/> YangSheng</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33548" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33548</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: fix wrong logic of stream allocation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 52116e1483cb24e3f5d0c2a3f20fa989dbd75e63</p> <p>are there any performance tests for this patch <a href="https://review.whamcloud.com/33195" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33195</a> ?</p> <p>Ihara had started running some tests on the patch, but I don't recall ever seeing the results.</p> <p>The main goal was to automate the original "manually reset to the start of the disk during benchmarking" behavior under normal usage. In particular, jump back to earlier groups when a bunch of free space becomes available, without having to continually scan the earlier groups for free space. The potential drawback is if this happens too frequently it could cause excessive seeking, but since it should only happen when there is a large amount of space any seek overhead should be smaller than the seek rate * IO size.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/21142/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21142/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: procfs entries for mballoc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 75703118588f2b23afd8c8815e5ebb768fc7a8ff</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34842" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34842</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: procfs entries for mballoc<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 1afff0d0a40ddf3c413c1db5a21d8d46da61e1c2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34842/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34842/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8365" title="Fix mballoc stream allocator to better use free space at start of drive" class="issue-link" data-issue-key="LU-8365">LU-8365</a> ldiskfs: procfs entries for mballoc<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: ea7103b0b1c360b0e6d7fe62e275df366bf4e31d</p> Duplicate LU-2377 Related LU-12970 LU-14438 LU-12103 Development Rank 1|hzygjr: Rank (Obsolete) 9223372036854775807 Severity