https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15002 Lustre <p>If LDISKFS target formatted with meta_bg option, then group descriptors are split across a target. Pre-reading optimization doesn't work for such metadata.</p> <p>With the current cluster size (no bigalloc option) a partition &gt;256TB can not be created without meta_bg, but there is a workaround.</p> <p>Filesystems can either be created using this new block group descriptor layout, or existing filesystems can be resized online, and the field s_first_meta_bg in the superblock will indicate the first block group using this new layout.</p> <p>The next steps allow to the creation of continuous group descriptors for the first 256TB and use meta_bg for all other groups.</p> <p>      1. Create &lt; 256 TB partition without the meta_bg flag</p> <p>      2. Extend the partition to the whole disk</p> <p>These steps can be done manually or mkfs can be modified.</p> LU-15002

Allocate metagroup descriptors continuously if possible

Improvement Major Resolved Fixed Dongyang Li Artem Blagodarenko Sat, 11 Sep 2021 17:18:10 +0000 Tue, 24 Oct 2023 01:39:26 +0000 Mon, 16 Oct 2023 12:50:46 +0000 Lustre 2.16.0 0 8 <p>With the <tt>sparse_super2</tt> option it is possible to avoid having multiple superblock and group descriptor backups, limiting the number of backups to 0, 1, or 2.</p> <p>However, even with this option, it is <b>not</b> possible to avoid enabling the <tt>meta_bg</tt> feature for filesystems larger than 256TiB, but I think it would be possible to fix this. </p> <p>The reason for this is that for filesystems above 256TiB, the primary group descriptor table in block group #0 (a 64-byte struct for each 128MiB group which stores the locations of the bitmaps and inode tables) itself consumes more than 128MiB (more than 2M block groups), and fills the whole first block group. At this point, any larger group descriptor would collide with the backup group descriptor, which is normally located in block group #1.</p> <p>I also tested with num_backup_sb=0 so that there would not be a backup superblock or backup group descriptor table, but this doesn't stop the enabling of <tt>meta_bg</tt> and its spread of the group descriptor blocks across the filesystem. It looks possible with relatively little effort to have <tt>mke2fs</tt> store the backup superblock(s) in later block groups (instead of group #1) that would otherwise have a backup under normal conditions (e.g. one of group #3, #5, #7, #9, #25, #27, #49, #81, #125, ...). </p> <p>While it is <em>possible</em> to arbitrarily locate the backup superblock and group descriptors in any group (there is the <tt>s_backup_bgs<span class="error">&#91;2&#93;</span></tt> field for storing their location), this would not be helpful if the primary superblock itself is corrupted. Sticking with "traditional" block groups for the superblock backups (powers of 3, 5, 7) makes it much easier for e2fsck and other tools to locate the backup in case the primary superblock is corrupted.</p> <p>Once <tt>sparse_super2</tt> moves the group descriptor backup out of group #1, then it should be possible for the group descriptor table to exceed 128MiB in size. Using group #5 for the backup would allow group descriptors for up to 1.25 PiB, while using group #9 would be good for up to 2.25 PiB, but this could be adjusted arbitrarily as the filesystem grows.</p> <p>The second <tt>sparse_super2</tt> backup is normally stored in the last full block group. For very large OSTs (&gt; 256TiB) the backup would not fit into that group, so should be located in the highest-numbered "traditional" <tt>sparse_super</tt> backup group number (there should be several choices given the large device size).</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/51295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/51295</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: allow selecting sparse_super2 backup<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: b0ede71742d759e6ec90964c50fa1a19f5835d23</p> <p>Two additional things need to be checked with the <tt>sparse_super2</tt> option:</p> <ul class="alternate" type="square"> <li>do we need to use the "<tt>-E packed_meta_blocks</tt>" option to get a good layout? This <em>should</em> be the default for <tt>sparse_super2</tt>, but I'm not positive.</li> <li>is the journal located at the start of the device or somewhere else? I vaguely recall that the journal was moved to the middle of the device to minimize average seek time on HDDs.</li> </ul> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dongyang" class="user-hover" rel="dongyang">dongyang</a><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hongchao.zhang" class="user-hover" rel="hongchao.zhang">hongchao.zhang</a> Would you please post the patches of E2fsprogs for this feature even they are not complete?</p> <p>Li Xi, the patch is tracked under <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/51295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/51295</a></p> <p>There still needs to be a second patch that updates e2fsck to automatically find the backup superblock and group descriptors from a sparse_super group number. See my previous notes on this. </p> <p>Andreas, finding the backup is added in patchset 4 of 51295.</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51723" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51723</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> utils: disable meta_bg and enable packed_meta_blocks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8f6c0f1af16390ff357d57c092aaacee043f1921</p> <p>It looks like filesystem resize is totally disallowed with the <tt>sparse_super2</tt> feature due to:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit b1489186cc8391e0c1e342f9fbc3eedf6b944c61 Author: Josh Triplett &lt;josh@joshtriplett.org&gt; AuthorDate: Mon Jun 7 12:15:24 2021 -0700 Commit: Theodore Ts'o &lt;tytso@mit.edu&gt; CommitDate: Thu Jun 24 10:22:36 2021 -0400 ext4: add check to prevent attempting to resize an fs with sparse_super2 The in-kernel ext4 resize code doesn't support filesystem with the sparse_super2 feature. It fails with errors like this and doesn't finish the resize: EXT4-fs (loop0): resizing filesystem from 16640 to 7864320 blocks EXT4-fs warning (device loop0): verify_reserved_gdb:760: reserved GDT 2 missing grp 1 (32770) EXT4-fs warning (device loop0): ext4_resize_fs:2111: error (-22) occurred during file system resize EXT4-fs (loop0): resized filesystem to 2097152 To reproduce: mkfs.ext4 -b 4096 -I 256 -J size=32 -E resize=$((256*1024*1024)) -O sparse_super2 ext4.img 65M truncate -s 30G ext4.img mount ext4.img /mnt python3 -c 'import fcntl, os, struct ; fd = os.open("/mnt", os.O_RDONLY | os.O_DIRECTORY) ; fcntl.ioctl(fd, 0x40086610, struct.pack("Q", 30 * 1024 * 1024 * 1024 // 4096), False) ; os.close(fd)' dmesg | tail e2fsck ext4.img The userspace resize2fs tool has a check for this case: it checks if the filesystem has sparse_super2 set and if the kernel provides /sys/fs/ext4/features/sparse_super2. However, the former check requires manually reading and parsing the filesystem superblock. Detect this case in ext4_resize_begin and error out early with a clear error message. Signed-off-by: Josh Triplett &lt;josh@joshtriplett.org&gt; Link: https://lore.kernel.org/r/74b8ae78405270211943cd7393e65586c5faeed1.1623093259.git.josh@joshtriplett.org Signed-off-by: Theodore Ts'o &lt;tytso@mit.edu&gt; </pre> </div></div> <p>I don't think this will be a huge problem for Lustre, since it is very rare to resize OSTs, but it would be good to backport that patch to ldiskfs so that the filesystem is not accidentally corrupted once <tt>sparse_super2</tt> is enabled by default. <del>This patch is already included in el7.9 and later</del>.</p> <p>It looks like it is not actually needed to <b>totally</b> disable online resize with <tt>sparse_super2</tt>. It looks like this is only a problem in the case where <tt>resize_inode</tt> is being used to reserve GDT blocks, because <tt>sparse_super2</tt> will store GDT backups in various different places. The <tt>resize_inode</tt> will not be enabled for filesystems &gt; 16TiB, so this should not actually be a problem under normal usage.</p> <p>It also looks possible to fix the online resize code to handle <tt>sparse_super2</tt> better by allowing <tt>resize_inode</tt> to store the GDT blocks after the current primary and two backup GDT copies for future resizing. That would ensure the resize kept all of the GDT blocks contiguous on disk, without having to e.g. move the block and inode bitmaps.</p> <p>One of the reasons that <tt>resize_inode</tt> is not used above 16TiB is because it is stored in block-mapped format, and that couldn't reserve blocks &gt; 2^32, but with <tt>sparse_super2</tt> the primary GDT and backup GDT #1 would always be near the start of the device with block numbers &lt; 2^32, and backup #2 GDT could be similarly constrained. The other reason that <tt>resize_inode</tt> wasn't used for huge filesystems is because the number of backup GDT blocks grows exponentially large as more backup groups are added to the filesystem, but this is also not a problem for <tt>sparse_super2</tt> since it only has one backup.</p> <p>I did some simple tests on 18k-03 system to verify if this patch can improve mke2fs performance as we expect. The results are shown as follows:</p> <p> </p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>mke2fs on OST0008<br/> (977TB)</th> <th class='confluenceTh'>time(s)</th> </tr> <tr> <td class='confluenceTd'>original<br/> (meta_bg,lazy_itable_init=1)</td> <td class='confluenceTd'>528</td> </tr> <tr> <td class='confluenceTd'>new<br/> (sparse_super2,packed_meta_blocks,lazy_itable_init=1)</td> <td class='confluenceTd'><b>37</b> <br/> (almost no difference w or w/ "nodiscard")</td> </tr> <tr> <td class='confluenceTd'>new<br/> (sparse_super2,packed_meta_blocks,lazy_itable_init=0)</td> <td class='confluenceTd'>too slow, about 10MB/s, unfinished--&gt; <br/> <b><font color="#172b4d">3.8TB/s</font> 260(s)</b> finished with the patch at <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52215" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52215</a></td> </tr> </tbody></table> </div> <p> </p> <p> </p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52215" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52215</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: batch zeroing inode table<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 37ac8287d28796446442e9c79f7b6827dceb08e9</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52219" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52219</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> e2fsck: check all sparse_super backups<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: fd0dd693eaf524fb50ed31288214e9a7a8a8648f</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52219/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52219/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> e2fsck: check all sparse_super backups<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: eb67ae2ec450eb7ea87d533bd24cc67340597fe6</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52273" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52273</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: set free blocks accurately for groups has GDT<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: afd1ed663877910ae8c63392830a3958b95e0afd</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52274" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52274</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: do not set the BLOCK_UNINIT on groups has GDT<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: 0fcb849941969f9553e43872800bd388689fced9</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52215/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52215/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: batch zeroing inode table<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: d7e9c047d4090abedc88e707da21b600640c70c3</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52273/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52273/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: set free blocks accurately for groups has GDT<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 6e18f9e2c1ae7300aade5f3b2d8b65b9f5e64fc3</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/52274/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/52274/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: do not set the BLOCK_UNINIT on groups has GDT<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 03a6a831ce5d78e024c03a21f7de88b14519ef99</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/tools/e2fsprogs/+/51295/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/tools/e2fsprogs/+/51295/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> mke2fs: try to pack the GDT blocks together<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: <br/> Commit: 27d5daaad86a70a54e579131b55b637c7e952cf5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51723/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51723/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15002" title="Allocate metagroup descriptors continuously if possible" class="issue-link" data-issue-key="LU-15002"><del>LU-15002</del></a> utils: disable meta_bg and enable packed_meta_blocks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7cce9f2d1c0911ee8501f08da6b6573735dee70e</p> <p>Landed for 2.16</p> Related LU-16750 Development Epic/Theme ldiskfs Rank 1|i0240n: Rank (Obsolete) 9223372036854775807