https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10520 Lustre <p>Taken from: <a href="https://github.com/intel-hpdd/intel-manager-for-lustre/issues/450" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/intel-hpdd/intel-manager-for-lustre/issues/450</a></p> <p> </p> <h3><a name="Description"></a>Description</h3> <p>When trying to create a MDT on a 29TB volume, the mke2fs command receive <tt>^extents</tt> instead of <tt>extents</tt> which cause the command to fails.</p> <h3><a name="Repro"></a>Repro</h3> <p> </p> <p>Create an MDT on a large LUN.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">modprobe osd_ldiskfs: 0 mkfs.lustre --mdt --mgsnode=172.21.61.200@tcp0 --mgsnode=172.21.61.206@tcp0 --failnode=172.21.61.200@tcp0 --reformat --index=0 --mkfsoptions=-I 512 -i 2048 -J size=2048 --backfstype=ldiskfs --fsname=BIGSI01 /dev/mapper/mpathb: 1 Permanent disk data: Target: BIGSI01:MDT0000 Index: 0 Lustre FS: BIGSI01 Mount type: ldiskfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=172.21.61.200@tcp:172.21.61.206@tcp failover.node=172.21.61.200@tcp device size = 30501008MB formatting backing filesystem ldiskfs on /dev/mapper/mpathb target name BIGSI01:MDT0000 4k blocks 7808258048 options -I 512 -i 2048 -J size=2048 -q -O dirdata,uninit_bg,^extents,mmp,dir_nlink,quota,huge_file,64bit,flex_bg -E lazy_journal_init -F mkfs_cmd = mke2fs -j -b 4096 -L BIGSI01:MDT0000 -I 512 -i 2048 -J size=2048 -q -O dirdata,uninit_bg,^extents,mmp,dir_nlink,quota,huge_file,64bit,flex_bg -E lazy_journal_init -F /dev/mapper/mpathb 7808258048 Found a gpt partition table in /dev/mapper/mpathb Extents MUST be enabled <span class="code-keyword">for</span> a 64-bit filesystem. Pass -O extents to rectify. mkfs.lustre FATAL: Unable to build fs /dev/mapper/mpathb (256) mkfs.lustre FATAL: mkfs failed 256 </pre> </div></div> <p>If I manually try the command and remove the "^" character, the command succeeds.</p> LU-10520

Cannot format large MDT with ldiskfs

Bug Blocker Resolved Fixed Yang Sheng Joe Grund Tue, 16 Jan 2018 17:48:02 +0000 Thu, 18 Apr 2019 22:27:24 +0000 Thu, 8 Mar 2018 17:40:00 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.11.0 0 9 <p>Ok found where the "^extends" come from :</p> <p>It's in build in the mkfs.lustre binary from lustre-2.10.2-1.src.rpm</p> <p>Exactly it seems to come from here:<br/> lustre/utils/mount_utils_ldiskfs.c:601: append_unique(anchor, ",", "^extents", NULL, maxbuflen);</p> <p>The line is the same for all the 2.10 releases, so this might not be entirely related to the issue (or else this probably would have shown in more places).</p> <p>Although, I am not sure that line 601 is supposed to look like that, as line 600 which add "uninit_bg" to the options list is :</p> <p>lustre/utils/mount_utils_ldiskfs.c:600: append_unique(anchor, ",", "uninit_bg", NULL, maxbuflen);</p> <p>Yang Sheng</p> <p>Could you please look into this?</p> <p>Thanks</p> <p>Peter</p> <p>There is no point in formatting an MDT filesystem larger than about 8-16TB, unless the new Data-on-MDT (DoM) feature is used, but that feature will not be available until the Lustre 2.11 release.</p> <p>Since the MDT (without the DoM feature) only holds inodes (1KB in size for 2.10 and later) plus directories, xattrs, and some Lustre log files (average 2KB per inode), and there is an upper limit of 4B inodes, 4B * 2KB = 8TB. Having a larger MDT is largely a waste of space, since the extra space above 8TB cannot be used until the DoM feature is available.</p> <p>If you are formatting this very large MDT in anticipation of the DoM feature, and are aware of this limitation that is OK. We need to make a patch to <tt>libmount_utils_ldiskfs.c</tt> to enable the <tt>extents</tt> feature only for MDT filesystems over 16TB in size.</p> <p>Hi,</p> <p>Thanks for the quick clarification.<br/> This is a test system and basically the MDT was built out of one of the LUN that was supposed to be an OST.<br/> There was no intention to use DoM at this point (but it is good to hear that the feature is coming).</p> <p>I was suspecting an issue with the size of the MDT as reducing the LUN from 29TB to 2TB allows it to format properly.</p> <p>Although I still get the weird "&#94;extents" in the mke2fs parameters list (is this a typo, or does '^' has special meaning ?).</p> <blockquote><p>mkfs.lustre --mdt --mgsnode=172.21.61.200@tcp0 --mgsnode=172.21.61.206@tcp0 --failnode=172.21.61.200@tcp0 --reformat --index=0 --mkfsoptions=-I 512 -i 2048 -J size=2048 --backfstype=ldiskfs --fsname=TEST01 /dev/mapper/mpathb: 0</p> <p>Permanent disk data:<br/> Target: TEST01:MDT0000<br/> Index: 0<br/> Lustre FS: TEST01<br/> Mount type: ldiskfs<br/> Flags: 0x61<br/> (MDT first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters: mgsnode=172.21.61.200@tcp:172.21.61.206@tcp failover.node=172.21.61.200@tcp</p> <p>device size = 1992294MB<br/> formatting backing filesystem ldiskfs on /dev/mapper/mpathb<br/> target name TEST01:MDT0000<br/> 4k blocks 510027366<br/> options -I 512 -i 2048 -J size=2048 -q -O dirdata,uninit_bg,^extents,mmp,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F<br/> mkfs_cmd = mke2fs -j -b 4096 -L TEST01:MDT0000 -I 512 -i 2048 -J size=2048 -q -O dirdata,uninit_bg,^extents,mmp,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/mapper/mpathb 510027366</p></blockquote> <p> </p> <p>Also last thing, creating a zpool of 29TB and creating an MDT on top of it works, even if as you mentioned, while running 2.10 this is mostly wasting space.</p> <p>Hi, Louis,</p> <p>The '^' means disabled feature. That said, we disable extents feature on MDT default.</p> <p>Thanks,<br/> YangSheng</p> <p>Note that formatting a ZFS MDT if 29 TB is not a necessarily a waste of space, since ZFS dynamically allocates inodes in the filesystem, though it uses about twice as much space per inode (4KB vs 2KB) compared to ldiskfs. That means if it is used to create a lot of files (as the MDT is traditionally used, it could hold about 7B inodes, or if it was used to hold 64KB files for DoM it could hold about 450M files. </p> <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31037" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31037</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10520" title="Cannot format large MDT with ldiskfs " class="issue-link" data-issue-key="LU-10520"><del>LU-10520</del></a> mkfs: enable extents for big MDT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e9ce401071c41b07d494291183d92f3beace24c4</p> <blockquote><p>There is no point in formatting an MDT filesystem larger than about 8-16TB, unless the new Data-on-MDT (DoM) feature is used, but that feature will not be available until the Lustre 2.11 release.</p></blockquote> <blockquote><p>Since the MDT (without the DoM feature) only holds inodes (1KB in size for 2.10 and later) plus directories, xattrs, and some Lustre log files (average 2KB per inode), and there is an upper limit of 4B inodes, 4B * 2KB = 8TB. Having a larger MDT is largely a waste of space, since the extra space above 8TB cannot be used until the DoM feature is available.</p></blockquote> <p>Hi,<br/> When calculating MDT disk space consumption, we might not forget space used by Changelog entries. As a rule of thumb, a Changelog entry is of an average of 125 B. And for instance, when touching a new file, if all Changelog entry types are recorded, 3 Changelog entries are generated (CREATE, OPEN, CLOSE).<br/> My point is space over 8TB on MDT device can be useful to store Changelog entries before they are consumed and cleared.</p> <p>Cheers,<br/> Sebastien.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31037/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31037/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10520" title="Cannot format large MDT with ldiskfs " class="issue-link" data-issue-key="LU-10520"><del>LU-10520</del></a> mkfs: enable extents for big MDT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eb65c3a586f1efd425f8360972b5d365cfecf7e1</p> <p>Landed for 2.11</p> Duplicate LU-12196 Related Development Rank 1|hzzr6n: Rank (Obsolete) 9223372036854775807 Severity