https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16162 Lustre <p>Disk performance degrades, when new blocks get allocated near the end of the disk,</p> <p>For example, the below are obdsurvey-results when mb_last_group is manually set to 0/75%/90% of max block group num before running the test:</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>3949.61 (0) vs 3677.15 (75%) vs 3133.43 (90%) [root@cslmo2305 ~]# rpm -qi lustre_ib | grep Version | dshbak -c ---------------- Version ---------------- 2.15.0.3_rc2_cray_165_g3355f27 [root@cslmo2305 ~]# echo 0 &gt; /proc/fs/ldiskfs/md*/mb_last_group [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 0 [root@cslmo2305 ~]# nobjlo=2 nobjhi=2 thrlo=1024 thrhi=1024 size=393216 rszlo=4096 rszhi=4096 tests_str="write read" obdfilter-survey | egrep -v "^done" 2&gt;/dev/null Wed May 18 13:23:57 UTC 2022 Obdfilter-survey for case=disk from cslmo2305 ost 1 sz 402653184K rsz 4096K obj 2 thr 1024 write 3949.61 [1399.72, 4272.29] read 4451.99 [1679.81, 5986.40] /usr/bin/iokit-libecho: line 235: 69223 Killed remote_shell $host "vmstat 5 &gt;&gt; $host_vmstatf" &amp;&gt; /dev/null [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 3565 [root@cslmo2305 ~]# echo 0 &gt; /proc/fs/ldiskfs/md*/mb_last_group [root@cslmo2305 ~]# nobjlo=2 nobjhi=2 thrlo=1024 thrhi=1024 size=393216 rszlo=4096 rszhi=4096 tests_str="write read" obdfilter-survey | egrep -v "^done" 2&gt;/dev/null Wed May 18 13:27:24 UTC 2022 Obdfilter-survey for case=disk from cslmo2305 ost 1 sz 402653184K rsz 4096K obj 2 thr 1024 write 3975.33 [1207.70, 4299.46] read 4517.36 [1623.78, 5675.05] /usr/bin/iokit-libecho: line 235: 76282 Killed remote_shell $host "vmstat 5 &gt;&gt; $host_vmstatf" &amp;&gt; /dev/null [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 3590 [root@cslmo2305 ~]# echo 1040830 &gt; /proc/fs/ldiskfs/md*/mb_last_group [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 1040830 [root@cslmo2305 ~]# nobjlo=2 nobjhi=2 thrlo=1024 thrhi=1024 size=393216 rszlo=4096 rszhi=4096 tests_str="write read" obdfilter-survey | egrep -v "^done" 2&gt;/dev/null Wed May 18 13:30:56 UTC 2022 Obdfilter-survey for case=disk from cslmo2305 ost 1 sz 402653184K rsz 4096K obj 2 thr 1024 write 3677.15 [2194.42, 3995.29] read 4819.83 [3596.32, 5391.15] /usr/bin/iokit-libecho: line 235: 82505 Killed remote_shell $host "vmstat 5 &gt;&gt; $host_vmstatf" &amp;&gt; /dev/null [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 1044021 [root@cslmo2305 ~]# echo 1040830 &gt; /proc/fs/ldiskfs/md*/mb_last_group [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 1040830 [root@cslmo2305 ~]# nobjlo=2 nobjhi=2 thrlo=1024 thrhi=1024 size=393216 rszlo=4096 rszhi=4096 tests_str="write read" obdfilter-survey | egrep -v "^done" 2&gt;/dev/null Wed May 18 13:34:34 UTC 2022 Obdfilter-survey for case=disk from cslmo2305 ost 1 sz 402653184K rsz 4096K obj 2 thr 1024 write 3666.51 [3231.40, 4047.68] read 4798.87 [3963.07, 5255.19] /usr/bin/iokit-libecho: line 235: 88607 Killed remote_shell $host "vmstat 5 &gt;&gt; $host_vmstatf" &amp;&gt; /dev/null [root@cslmo2305 ~]# cat /proc/fs/ldiskfs/md*/mb_last_group 1044030 [root@cslmo2305 ~]# echo 1248996 &gt; /proc/fs/ldiskfs/md </pre> </div></div></blockquote> LU-16162

ldiskfs: use low disk tracks for block allocation on empty or moderately full filesystems.

Improvement Major Open Unresolved Alexander Zarochentsev Alexander Zarochentsev ldiskfs Thu, 15 Sep 2022 14:11:54 +0000 Wed, 10 May 2023 22:57:26 +0000 0 4 <p>I've been thinking about this issue for some time already, and I think it makes sense to use the <a href="https://jira.whamcloud.com/browse/LU-14438" title="backport ldiskfs mballoc patches" class="issue-link" data-issue-key="LU-14438">LU-14438</a> patches as a starting point for this. That patch provides an array of allocation groups sorted by size of free extent, and the array is used when searching for a new group for allocations. </p> <p>To provide the start/end segregation needed to isolate the slower tracks of the disk, a threshold could be set (eg. 80% of groups, or a specific group number), and this could be used to efficiently split the groups into two arrays. The "fast" array, for groups below the threshold, and the "slow" array for groups larger than the threshold. Allocations would prefer groups from the fast array if there are suitable free chunks, and only look for groups in the slow array of there were none in the fast array. </p> <p>That would only be a small change to the mballoc code, as well as an O(1) change to the array insertion code to pick the correct array in which to insert each group. </p> <p>If there was some way for clients to specify QOS for files ("<tt>lfs ladvise</tt>" or with "<tt>lfs migrate</tt>"?), it would even be possible to have osd-ldiskfs allocate objects into the slow groups directly. </p> <p>"Alexander Zarochentsev &lt;alexander.zarochentsev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48558" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48558</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16162" title="ldiskfs: use low disk tracks for block allocation on empty or moderately full filesystems." class="issue-link" data-issue-key="LU-16162">LU-16162</a> ldiskfs: keep low tracks allocated by mballoc<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 3<br/> Commit: cfdf70dc5b3aad26ff746f20ee030c389f9a7715</p> <blockquote> <p>If there was some way for clients to specify QOS for files ("lfs ladvise" or with "lfs migrate"?), it would even be possible to have osd-ldiskfs allocate objects into the slow groups directly.</p></blockquote> <p>Shuichi had a very good use case for being able to write into the "slow" part of the filesystem. When migrating data from an old filesystem to a new filesystem, the old data will be copied into the newly-formatted OSTs, and fill all of the low groups (high bandwidth), leaving the new data to use slower parts of the disk. It would be useful to have some mechanism (ladvise process setting, environment variable, layout, or <tt>fcntl(F_SET_RW_HINT)</tt>?) to force object allocation to e.g. the last 30% of groups for cases like this, so that the beginning of the filesystem remains available for new usage.</p> <p>&gt; if there was some way for clients to specify QOS for files ("lfs ladvise" or with "lfs migrate"?), it would even be possible to have osd-ldiskfs allocate objects into the slow groups directly.</p> <p>no, the patch only addresses a simple but annoying case when mb_last_group points to some high block group num but the fs still almost empty (due to repeatable write / delete usage pattern) also <a href="https://jira.whamcloud.com/browse/LU-15319" title="Weird mballoc behaviour" class="issue-link" data-issue-key="LU-15319"><del>LU-15319</del></a> is about a weird mballoc optimization causing skipping of already initialized block groups , i.e. ldiskfs fs starts to write to not initialized part, after some iterations, fs users would notice an empty fs slowness.</p> Related LU-14438 LU-15319 Development Rank 1|i030an: Rank (Obsolete) 9223372036854775807