https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14438 Lustre <p>There is an upstream patch series that is adding improved mballoc handling for efficiently finding suitable allocation groups in a filesystem. In particular, patch <br/> <a href="https://patchwork.ozlabs.org/project/linux-ext4/patch/20210209202857.4185846-5-harshadshirwadkar@gmail.com/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.ozlabs.org/project/linux-ext4/patch/20210209202857.4185846-5-harshadshirwadkar@gmail.com/</a> "<tt>ext4: improve cr 0 / cr 1 group scanning</tt>" is the important part of the series. </p> LU-14438

backport ldiskfs mballoc patches

Improvement Minor Open Unresolved Artem Blagodarenko Andreas Dilger ldiskfs Tue, 16 Feb 2021 17:16:37 +0000 Thu, 28 Sep 2023 02:53:15 +0000 2 18 <p>I've attached v2 of the <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/37510/37510_ext4-improve-cr-0-cr-1-group-scanning-v2.patch" title="ext4-improve-cr-0-cr-1-group-scanning-v2.patch attached to LU-14438">ext4-improve-cr-0-cr-1-group-scanning-v2.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> from the list (against current Linux master, not ported to any RHEL kernel yet). While there is still work being done to improve this patch, I think it would be useful to see how much this improves performance for a large <b>fragmented</b> filesystem and/or hurts performance for a large empty filesystem. Having some performance feedback earlier would allow improving the patch before it is included in the upstream kernel, and if it shows good promise I think it is a better long-term solution than the current <tt>ext4-simple-blockalloc.patch</tt> that we are carrying, since that patch just <em>reduces</em> the number of times useless groups are scanned but doesn't avoid sequential scanning completely like this new patch does.</p> <p>I believe for testing purpose and later, after successful <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/37510/37510_ext4-improve-cr-0-cr-1-group-scanning-v2.patch" title="ext4-improve-cr-0-cr-1-group-scanning-v2.patch attached to LU-14438">ext4-improve-cr-0-cr-1-group-scanning-v2.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> testing, ext4-simple-blockalloc.patch should be dropped completely because it makes porting difficult. </p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/43232" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43232</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14438" title="backport ldiskfs mballoc patches" class="issue-link" data-issue-key="LU-14438">LU-14438</a> ldiskfs: improvements to mballoc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b7e2d9466f2a45d3c9a687cf06155d4e75b020c9</p> <p>The new mballoc patch from the upstream kernel keeps an array with 2^n order of the largest range of free blocks in the group (between 2^0=1 free block and 2^16 = 32768 free blocks), and puts each group into the appropriate list after each alloc/free. It uses round-robin selection for groups in the per-order list, so it is still possible to get into situations similar to <tt>mb_last_group</tt> being very large, where the allocations are done at the end of the filesystem (lower bandwidth) even though there are many groups with free space available at the start of the filesystem (higher bandwidth).</p> <p>It would make sense to enhance the new allocator to have <b>two</b> per-order lists for tracking the free blocks (on HDD OSTs at least, based the "rotational" parameter of the block device) - one list for the groups in the first ~70% of the filesystem that have good performance and a second list for groups in the last ~30% of the filesystem that have lower performance. Groups in the second list would only be used if there are no free groups of the right order in the first list. That would bias allocations to the start of the device so that it avoids needless slowdowns when the filesystem is not full. Since the amount of memory used for the per-order array itself is small (array of 17 pointers), and it is easy to decide which array a given group is put into based on the group number this would not increase allocation overhead. That would probably be much more efficient than trying to keep the groups within each order totally sorted.</p> <p>Andreas,</p> <p>Regarding the comment in the code patch: "<em>the groups may not get traversed linearly. That may result in subsequent allocations being not close to each other. And so, the underlying device may get filled up in a non-linear fashion.</em>"... rather than using a fixed MB_DEFAULT_LINEAR_LIMIT what do you think about using something like the following algorithm? (I have been playing with external setting of mb_last_group to an "optimal" value, but incorporating the idea into ext4 would be much cleaner.)</p> <p>The general idea is to work <b>backwards</b> through mb_groups info and use a "decay" algorithm to determine an <b>adjusted</b> value for free block count for each group. Then set mb_last_group based on the largest adjusted block count value. Attached is a simple script <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/40818/40818_pick_mb_last_group.sh" title="pick_mb_last_group.sh attached to LU-14438">pick_mb_last_group.sh^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> to demonstrate the approach. In my limited testing, it does seem to pick a group number that not only has a large "bfree" value but is also followed by other groups that generally have large-ish bfree values as well.<br/> Obviously more cleanup on the script would be needed to make it "production ready", and some theory and testing applied to set a good $decay value, and there is also a question of how often to run the tool and change the value... but hopefully the script at least clarifies the approach. Further enhancements could include a check of "/sys/block/${dev}/queue/rotational"; if a device is spinning rust, then adjust the weighted score further with a penalty for higher group numbers.</p> <p>Thanks,<br/> Nathan</p> <p>There are additional patches to fix the mballoc mb_optimized_scan=1 use case:<br/> <a href="https://patchwork.ozlabs.org/project/linux-ext4/list/?series=317391" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.ozlabs.org/project/linux-ext4/list/?series=317391</a></p> <p>These fix a number of sub-optimal allocation decisions in the earlier patches.</p> <p>I've filed <a href="https://jira.whamcloud.com/browse/LU-16155" title="allow importing inode/block allocation maps to new ldisks filesystem" class="issue-link" data-issue-key="LU-16155">LU-16155</a> to enhance debugfs to allow "importing" the block and inode allocation maps into a newly-formatted filesystem to simplify testing of this problem. We could collect the debugfs information from real filesystems that are having allocation performance issues as needed in order to test changes to mballoc.</p> <p>I've tried to port some of the upstream mballoc patches in this, but it looks too big for a single patch.</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51472" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51472</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14438" title="backport ldiskfs mballoc patches" class="issue-link" data-issue-key="LU-14438">LU-14438</a> ldiskfs: backport ldiskfs mballoc patches<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2439579001a928714a640ec469a2d833ea5e8337</p> <p>There are cases where we may want to make empty filesystem performance <b>worse</b>, but the 90% performance better. We could use the new mballoc array lists to spread out allocations across the disk more evenly. </p> <p>I had previously considered that we might split groups into two arrays (as we are doing with IOPS groups in <a href="https://jira.whamcloud.com/browse/LU-16750" title="optimize ldiskfs internal metadata allocation for hybrid storage LUNs" class="issue-link" data-issue-key="LU-16750">LU-16750</a>) 80% at the start of the disk and 20% at the end of the disk (or 90/10%) so groups at end of the filesystem are only used when the first groups are mostly full. However, this would mean that performance would suddenly drop once the filesystem hit 80% full.</p> <p>We could instead do things like split the groups into eg. 16 separate arrays by offset, and then have a clock that rotates allocations around the regions eg. every second, so that groups are not used start-to-end during allocation. We would still want <em>some</em> locality in allocations, so we are not seeking wildly around the disk for files being written concurrently, but are always using the end of the disk some fraction of the time. This would hopefully even out the performance over the filesystem lifetime for uses that demand more consistent performance instead of "best possible".</p> <p>We could even hint via "<tt>lfs ladvise</tt>" and/or "<tt>ionice</tt>" for a file or process to force all file allocations to the slow part of the disk for cases of archiving old files. I don't think it makes sense to allow "improving" allocations because everyone would want that and it would be no different than today. </p> Duplicate Related LU-8365 LU-15319 LU-17153 LU-16162 LU-16750 LU-12970 LU-16155 LU-14305 Development Rank 1|i01mqv: Rank (Obsolete) 9223372036854775807