https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-255 Lustre <p>There are a number of ext4 features that we should be enabling by default for newly-formatted ldiskfs filesystems. In particular, the flex_bg option is important for reducing e2fsck time as well as avoiding "slow first write" issues that have hit a number of customers with fuller OSTs. Using flex_bg would avoid 10-minute delay at mount time or for each e2fsck run. As well, it would be useful to also enable other features like huge_file (files &gt; 2TB) and dir_nlink (&gt; 65000 subdirectories) by default.</p> <p>All of these features are enabled by default if we format the filesystem with the option "-t ext4". Alternately, we could enable these individually in enable_default_backfs_features().</p> <p>See <a href="http://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_fujita.pdf" class="external-link" target="_blank" rel="nofollow noopener">http://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_fujita.pdf</a> for a summary of improvements. While we won't see the 12h e2fsck -&gt; 5 minute e2fsck improvement shown there (we already use extents and uninit_bg), the flex_bg feature is definitely still a win.</p> LU-255

use ext4 features by default for newly formatted filesystems

Improvement Major Closed Fixed Andreas Dilger Andreas Dilger Fri, 29 Apr 2011 12:36:28 +0000 Thu, 19 May 2011 01:00:44 +0000 Thu, 19 May 2011 01:00:44 +0000 Lustre 2.1.0 Lustre 2.1.0 0 5 <p>I'll be running some testing with ~8 TB and larger LUNs over the next few weeks to see the performance impacts of various settings for the groups in a flexible block group, when I have some results I will post them here. My main focus though is to alleviate the slow mounts and issues from <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a>. At least mkfs.lustre for a 9 TB LUN drops from 17 minutes to 6 minutes with of &gt;64 for the number of groups. </p> <p>Andreas seems to be working on this</p> <p>Jeremy, test RPMs are available via <a href="http://review.whamcloud.com/#change,480" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,480</a> if you are able to test them. They are built from the lustre-release repository, so the mkfs.lustre is not directly useful to you if you are testing on 1.8.x.</p> <p>The default parameters for an OST with this patch (assuming a large-enough LUN size and ext4-based ldiskfs) are:</p> <p>mke2fs -j -b 4096 -L lustre-OSTffff -J size=400 -I 256 -i 262144 -O extents,uninit_bg,dir_nlink,huge_file,flex_bg -G 256 -E resize=4290772992,lazy_journal_init, -F </p> {dev}<br/> <br/> For an MDT they are:<br/> <br/> mke2fs -j -b 4096 -L lustre-MDTffff -J size=400 -I 512 -i 2048 -O dirdata,uninit_bg,dir_nlink,huge_file,flex_bg -E lazy_journal_init, -F {dev} <p>Oleg, this patch should be included into the 2.1 release - it dramatically speeds up mkfs and should fix (for new filesystems) the slow startup problems seen in <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a>.</p> <p>I'm also interested in this patches and just tested patched RPMs. When I formatted the MDT (16TB), it failed due to the following errors. Any advises? OST format worked well.</p> <ol> <li>mkfs.lustre --verbose --reformat --mgs --mdt /dev/mpath/mdt</li> </ol> <p> Permanent disk data:<br/> Target: lustre-MDTffff<br/> Index: unassigned<br/> Lustre FS: lustre<br/> Mount type: ldiskfs<br/> Flags: 0x75<br/> (MDT MGS needs_index first_time update )<br/> Persistent mount opts: user_xattr,errors=remount-ro<br/> Parameters:</p> <p>device size = 14934016MB<br/> formatting backing filesystem ldiskfs on /dev/mpath/mdt<br/> target name lustre-MDTffff<br/> 4k blocks 3823108096<br/> options -J size=400 -I 512 -i 2048 -O dirdata,uninit_bg,dir_nlink,huge_file,flex_bg -E lazy_journal_init, -F<br/> mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDTffff -J size=400 -I 512 -i 2048 -O dirdata,uninit_bg,dir_nlink,huge_file,flex_bg -E lazy_journal_init, -F /dev/mpath/mdt 3823108096<br/> cmd: mke2fs -j -b 4096 -L lustre-MDTffff -J size=400 -I 512 -i 2048 -O dirdata,uninit_bg,dir_nlink,huge_file,flex_bg -E lazy_journal_init, -F /dev/mpath/mdt 3823108096<br/> mke2fs 1.41.12.2.ora1 (14-Aug-2010)<br/> mke2fs: too many inodes (7646216192), raise inode ratio?</p> <p>mkfs.lustre FATAL: Unable to build fs /dev/mpath/mdt (256)</p> <p>mkfs.lustre FATAL: mkfs failed 256</p> <p>Formatting MDT also worked, when I added --mkfsoptions="-i 4096" to mkfs.lustre...</p> <p>Ihara, thanks for testing. Did you teat on 2.x or 1.8? </p> <p>As for the problem hit on the MDT, I agree that the mkfs.lustre command should handle this case better. However, I also think that it doesn't make sense to have a 16TB MDT because that much space will never be used. One of the changes being made in this patch is to reduce the default inode ratio to 2048 bytes per inode, which is still very safe but allows more inodes for a given LUN size. I would recommend simply using a smaller LUN for the MDT. With the new inode ratio 8TB is enough for the maximum 4B inodes. </p> <p>I wonder how safe is it to not zero the journal? Suppose this is mkfs on top of previous ext4. Could it happen then that in certain cases old transactions from the journal would be picked up?</p> <p>Realistically, it is very unlikely to re-use anything from the internal journal in this case. The journal superblock will be rewritten, with a new journal transaction ID of 1, and marking no oustanding transactions to recover, and when it is mounted the TID will increment from 1.</p> <p>If the node crashed before it had overwritten the journal (unlikely even under relatively low usage) there would still need to be transactions left in the journal that aligned right after the end of the current transaction, and also with the next TID in sequence.</p> <p>In practice I think the chance of this is very low except in test filesystems that are reformatted repeatedly after a very short lifespan, but if you want I could drop this part of the patch. It avoids 400MB of IO to the device at mke2fs time, but even then this is a small portion of the inode table blocks being written.</p> <p>I'm testing on 2.x. (got RPMs from <a href="http://review.whamcloud.com/#change,480" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,480</a>) there are some test updates. We have a 8TB (changed size from 16TB) MDT and 16TB OSTs, here is time for mkfs.lustre.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> un-patched(sec) patched(sec) MDT 3591 3361 OST 1836 15 </pre> </div></div> <p>Formatting the OSTs was dramatically speedup, but didn't see big acceleration of formatting MDT.</p> <p>I suspect that the MDT format time is actually more than 2x as fast <em>per_inode</em>, because it is writing 2x as many inodes for the same amount of space (using "-i 2048" for patched, and "-i 4096" for unpatched). Even if it isn't running mke2fs faster on the MDT, it should also be running e2fsck faster due to flex_bg.</p> <p>ah, yes. is it worth to test "-i 2048" on un-patched to make sure speedup? And, I'm going to test e2fsck to MDT and OST (in 0%, 50%, 80% usage case) on un-patched and patched. </p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/utils/mkfs_lustre.c</li> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=ofa/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,ofa #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/utils/mkfs_lustre.c</li> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> <li>lustre/utils/mkfs_lustre.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/122/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #122</a><br/> <a href="https://jira.whamcloud.com/browse/LU-255" title="use ext4 features by default for newly formatted filesystems" class="issue-link" data-issue-key="LU-255"><del>LU-255</del></a>: enable ext4 features by default</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=eb012d4a10208b26c2d3e795a90f1bb07dde6d91" class="external-link" target="_blank" rel="nofollow noopener">eb012d4a10208b26c2d3e795a90f1bb07dde6d91</a><br/> Files : </p> <ul> <li>lustre/utils/mkfs_lustre.c</li> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Patch is landed for 2.1.0, closing bug.</p> Development Rank 1|hzv99j: Rank (Obsolete) 5035