https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16305 Lustre <p>Attempting to mkfs.lustre a disk below 16TiB in size but above the value computed for "resize" by mkfs.lustre (for 4KiB blocks, 32GiB shy of 16TiB) results in a failure as mke2fs requires resize to be greater than the specified capacity.</p> <p><tt>Example:</tt><br/> <tt>mkfs.lustre --ost --reformat --servicenode &lt;elided&gt; --fsname=lustrefs --index 1 --mgsnode &lt;elided&gt; --backfstype=ldiskfs /dev/ost1</tt><br/> <tt>mkfs.lustre FATAL: Unable to build fs /dev/ost1 (256)</tt><tt>mkfs.lustre FATAL: mkfs failed 256</tt><tt>   Permanent disk data:</tt><br/> <tt>Target:     lustrefs:OST0001</tt><br/> <tt>Index:      1  </tt><br/> <tt>Lustre FS:  lustrefs</tt><br/> <tt>Mount type: ldiskfs</tt><br/> <tt>Flags:      0x1062</tt><br/> <tt>              (OST first_time update no_primnode )</tt><br/> <tt>Persistent mount opts: ,errors=remount-ro</tt><br/> <tt>Parameters: failover.node=&lt;elided&gt; mgsnode=&lt;elided&gt;</tt><tt>device size = 16777152MB</tt><br/> <tt>formatting backing filesystem ldiskfs on /dev/ost1</tt><br/> <tt>  target name   lustrefs:OST0001</tt><br/> <tt>  kilobytes     17179803648</tt><br/> <tt>  options         -J size=1024 -I 512 -i 524288 -q -O extents,uninit_bg,mmp,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E resize=\"4290772992\",lazy_journal_init=\"0\",lazy_itable_init=\"0\" -F</tt><br/> <tt>mkfs_cmd = mke2fs -j -b 4096 -L lustrefs:OST0001   -J size=1024 -I 512 -i 524288 -q -O extents,uninit_bg,mmp,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E resize=\"4290772992\",lazy_journal_init=\"0\",lazy_itable_init=\"0\" -F /dev/ost1 17179803648k</tt><br/> <tt>   detected raid stride 4194304 too large, use optimum 512</tt><br/> <tt>   detected raid stripe width 67108864 too large, use optimum 512</tt><br/> <tt>   The resize maximum must be greater than the filesystem size.</tt><br/> <tt>   </tt><br/> <tt>   </tt><br/> <tt>   Bad option(s) specified:</tt><tt>   Extended options are separated by commas, and may take an argument which</tt><br/> <tt>    is set off by an equals ('=') sign. </tt><tt>   Valid extended options are:</tt><br/> <tt>    mmp_update_interval=&lt;interval&gt;</tt><br/> <tt>    num_backup_sb=&lt;0|1|2&gt;</tt><br/> <tt>    stride=&lt;RAID per-disk data chunk in blocks&gt;</tt><br/> <tt>    stripe-width=&lt;RAID stride * data disks in blocks&gt; </tt><br/> <tt>    offset=&lt;offset to create the file system&gt;</tt><br/> <tt>    resize=&lt;resize maximum size in blocks&gt;</tt><br/> <tt>    packed_meta_blocks=&lt;0 to disable, 1 to enable&gt;</tt><br/> <tt>    lazy_itable_init=&lt;0 to disable, 1 to enable&gt;</tt><br/> <tt>    lazy_journal_init=&lt;0 to disable, 1 to enable&gt;</tt><br/> <tt>    root_owner=&lt;uid of root dir&gt;:&lt;gid of root dir&gt;</tt><br/> <tt>    test_fs</tt><br/> <tt>    discard</tt><br/> <tt>    nodiscard</tt><br/> <tt>    encoding=&lt;encoding&gt;</tt><br/> <tt>    encoding_flags=&lt;flags&gt;</tt><br/> <tt>    quotatype=&lt;quota type(s) to be enabled&gt;</tt></p> LU-16305

mkfs.lustre fails on devices between 16TiB-32GiB and 16TiB-1B

Bug Minor Open Unresolved Ellis Wilson Ellis Wilson Wed, 9 Nov 2022 16:47:06 +0000 Sun, 13 Nov 2022 20:26:48 +0000 0 4 <p>Ellis, thanks for filing the ticket. Until this is fixed in the code, it should be possible to work around the issue by adding "<tt>,^resize_inode</tt>" to the "<tt>-O</tt>" feature list, and removing "<tt>resize=4290772992,</tt>" from the "<tt>-E</tt>" extended options list on the <tt>mkfs.lustre</tt> command line.</p> <p>No problem!  I fixed it with the following patch (going through the mechanics presently to test/submit the patch):</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--- a/lustre/utils/libmount_utils_ldiskfs.c +++ b/lustre/utils/libmount_utils_ldiskfs.c @@ -885,6 +885,15 @@ int ldiskfs_make_lustre(struct mkfs_opts *mop)                 append_unique(start, ext_opts ? "," : " -E ",                               "resize", buf, maxbuflen);                 ext_opts = 1; + +               /* The resize maximum must be greater than filesystem size, but for disks +                * or arrays just shy of 16TiB you can get into a situation where capacity +                * is between resize_blks and 16TiB.    Shrink the drive size to 1MiB less +                * than resize in these scenarios (at most ~0.1% capacity is lost).  +                */ +               if (resize_blks &lt;= mop-&gt;mo_device_kb / mop-&gt;mo_blocksize_kb) { +                       mop-&gt;mo_device_kb = (long long)(resize_blks) * (long long)mop-&gt;mo_blocksize_kb - 1024; +               }         }           /* Avoid zeroing out the full journal - speeds up mkfs */ {{</pre> </div></div> <p>As posted on lustre-discuss, I have some questions about the intent behind resize, and IDK how to get this assigned to me (maybe that's only for WC people).</p> <p>Rather than shrink the MDT device, it would be better to just disable the <tt>resize_inode</tt> feature for such filesystems, since it is not useful for filesystems over 16TiB anyway.</p> <p>I believe this only applies to OSTs, and while I can disable it I'd like to better understand what the optimization is attempting to accomplish first.  I think you put this block in around 2011 (could totally be wrong – it's moved around a few times).  Do you remember what it was accomplishing?  I'm really struggling to understand this comment block:<br/>    871   /* In order to align the filesystem metadata on 1MB boundaries,<br/>    872    * give a resize value that will reserve a power-of-two group<br/>    873    * descriptor blocks, but leave one block for the superblock.<br/>    874    * Only useful for filesystems with &lt; 2^32 blocks due to resize<br/>    875    * limitations.</p> <p>Is ext metadata really unaligned without specifying resize?  Some docs suggest that without giving this, mke2fs plans for up to 1024 times the original size of the filesystem, so I don't feel like this is a case where we're trying to plan ahead more than mke2fs already does.</p> <p>The <tt>resize_inode</tt> feature only works up to 16TB, so it is basically useless for the problematic filesystem and may as well be disabled for such filesystems. There is a different feature (<tt>meta_bg</tt>) that is used for resizing filesystems beyond 16TB. The 1024x resize is based on a starting filesystem size that is much smaller. </p> <p>Yes, the ext4 metadata is not aligned to 1MB boundaries by default, and this option (along with some others added in the same patch) ensures that other metadata was located with proper 1MB alignment for HDD RAID alignment. That is not so important for flash MDTs at this point either.</p> <p>So my approach to fixing this issue would be to disable the <tt>resize_inode</tt> feature (if this isn't done automatically already) and not specify the "<tt>-E resize=nnnn</tt>" option for filesystems that are close to 16TB in size.</p> <p>Thanks for the clarification Andreas.  I've revised my in-house fix, and will run it through the steps on your submitting changes wiki shortly.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=elliswilson" class="user-hover" rel="elliswilson">elliswilson</a> I have added you to the developers group for the community project so you should now be able to do things like assign tickets to yourself etc</p> Development Epic/Theme ldiskfs Rank 1|i0356v: Rank (Obsolete) 9223372036854775807 Severity