https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9788 Lustre <p>ORNL's main production file system was formatted during the 2.4.3 lustre time frame. Since then we have move to 2.5 and now to lustre 2.8.0 without updating the ldiskfs on line format. This ticket is a request into what has changed and the impact of the changes. Lastly we need to ensure the upgrade it correct when done.</p> LU-9788

upgrading ldiskfs on-disk format from 2.4.3 lustre version to 2.8.0

Question/Request Major Resolved Done Andreas Dilger James A Simmons Thu, 20 Jul 2017 13:59:27 +0000 Sat, 10 Mar 2018 07:57:57 +0000 Sat, 10 Mar 2018 07:57:57 +0000 0 6 <p>Andreas</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>We work hard to maintain upgrade and downgrade compatibility between Lustre releases for the on-disk format. New features that affect the on-disk format in a manner that prevents a downgrade to the previous Lustre version will typically require explicit action from the administrator to enable before it is used, to allow the system to be upgraded without affecting the disk format, and only enabling the new feature once the new Lustre release is known to be stable in your environment. </p> <p>It would be good to get the output of "dumpe2fs -h" from the MDT and one OST (assuming they are the same) to see what ldiskfs features are currently enabled, and check if there may be performance improvements possible after the upgrade. </p> <p>Secondly, in addition to upgrading the servers, will you also be upgrading the clients, or will you be running with different client versions?  I believe that you may already be running 2.8 clients on your system. </p> <p>There are two issues that I'm aware of that would affect upgrade+downgrade to a 2.4 MDS. One is that the client multiple metadata request feature ("multi-slot last_rcvd") sets a flag on the MDT for the new recovery file format that prevents mounting on an unsupported version of Lustre (<a href="https://jira.whamcloud.com/browse/LU-7410" title="After downgrade from 2.8 to 2.5.5, hit unsupported incompat filesystem feature(s) 400" class="issue-link" data-issue-key="LU-7410"><del>LU-7410</del></a>). This has a simple workaround if it is hit if you need to downgrade, as described in that ticket. </p> <p>The second is related to LFSCK (<a href="https://jira.whamcloud.com/browse/LU-8605" title="Downgrading from 2.8 to 2.5: fsck_namespace_load: fail to load lfsck_namespace" class="issue-link" data-issue-key="LU-8605"><del>LU-8605</del></a>), and can be avoided by having the fix applied to your 2.8 code before the upgrade. You may already have a fix for this issue. </p> <p>If you want to be prudent, it makes sense to create a backup of the MDT prior to the upgrade. This can be done with "dd"of the raw MDT filesystem to.a backup device before installing the new Lustre release. DDN has also been testing the use of "e2image" to make copies of the ldiskfs metadata, which have the advantage of only backing up the in-use parts of the device, and are stored more compactly. It would be possible to make an e2image backup of the OSTs as well, since the actual space used would be relatively small. <br/>  </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,</p> <p>Does this answer your question(s)?</p> <p>Regards,</p> <p>Brad</p> <p>I need to talk to Andreas in detail about this.</p> <p>Both our clients and server back end are running lustre 2.8.1. Its just the ldiskfs format that hasn't been upgraded since our 2.5 days. Okay here is the debugfs output from our MDS server:</p> <p><span class="error">&#91;root@atlas1-mds1 ~&#93;</span># dumpe2fs -h /dev/mapper/atlas1-mdt1<br/> dumpe2fs 1.42.13.wc5 (15-Apr-2016)<br/> Filesystem volume name: atlas1-MDT0000<br/> Last mounted on: /<br/> Filesystem UUID: 182b7f08-b0ff-4803-aa46-c110ac95acfe<br/> Filesystem magic number: 0xEF53<br/> Filesystem revision #: 1 (dynamic)<br/> Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery flex_bg ea_inode dirdata sparse_super large_file huge_file uninit_bg dir_nlink quota<br/> Filesystem flags: signed_directory_hash <br/> Default mount options: user_xattr<br/> Filesystem state: clean<br/> Errors behavior: Continue<br/> Filesystem OS type: Linux<br/> Inode count: 1073741824<br/> Block count: 536870912<br/> Reserved block count: 26843545<br/> Free blocks: 272442789<br/> Free inodes: 682054233<br/> First block: 0<br/> Block size: 4096<br/> Fragment size: 4096<br/> Reserved GDT blocks: 1024<br/> Blocks per group: 16384<br/> Fragments per group: 16384<br/> Inodes per group: 32768<br/> Inode blocks per group: 4096<br/> Flex block group size: 16<br/> Filesystem created: Tue Oct 1 11:02:25 2013<br/> Last mount time: Tue Aug 22 12:14:37 2017<br/> Last write time: Tue Aug 22 12:14:37 2017<br/> Mount count: 4<br/> Maximum mount count: -1<br/> Last checked: Tue Jun 20 08:52:59 2017<br/> Check interval: 0 (&lt;none&gt;)<br/> Lifetime writes: 46 TB<br/> Reserved blocks uid: 0 (user root)<br/> Reserved blocks gid: 0 (group root)<br/> First inode: 11<br/> Inode size: 512<br/> Required extra isize: 28<br/> Desired extra isize: 28<br/> Journal UUID: 33c503b6-30a3-481b-8ac6-121e2144bb79<br/> Journal device: 0xfd06<br/> Default directory hash: half_md4<br/> Directory Hash Seed: bfef46fe-273d-43ea-95e3-d9ebcd11a516<br/> Journal backup: inode blocks<br/> User quota inode: 3<br/> Group quota inode: 4</p> <p>and here is the output from one of our OSS servers:</p> <p><span class="error">&#91;root@atlas-oss1a1 ~&#93;</span># dumpe2fs -h /dev/mapper/atlas-ddn1a-l0 <br/> dumpe2fs 1.42.13.wc5 (15-Apr-2016)<br/> Filesystem volume name: atlas1-OST0000<br/> Last mounted on: /<br/> Filesystem UUID: f28766dc-9235-45e4-9ddc-f502ad57276c<br/> Filesystem magic number: 0xEF53<br/> Filesystem revision #: 1 (dynamic)<br/> Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent mmp flex_bg sparse_super large_file huge_file uninit_bg dir_nlink quota<br/> Filesystem flags: signed_directory_hash <br/> Default mount options: user_xattr acl<br/> Filesystem state: clean<br/> Errors behavior: Continue<br/> Filesystem OS type: Linux<br/> Inode count: 29343744<br/> Block count: 3755999232<br/> Reserved block count: 187799961<br/> Free blocks: 1333048954<br/> Free inodes: 28069821<br/> First block: 0<br/> Block size: 4096<br/> Fragment size: 4096<br/> Reserved GDT blocks: 127<br/> Blocks per group: 32768<br/> Fragments per group: 32768<br/> Inodes per group: 256<br/> Inode blocks per group: 16<br/> Flex block group size: 256<br/> Filesystem created: Tue Oct 1 11:02:26 2013<br/> Last mount time: Tue Aug 22 12:14:42 2017<br/> Last write time: Tue Aug 22 12:14:42 2017<br/> Mount count: 23<br/> Maximum mount count: -1<br/> Last checked: Sun Feb 7 16:31:09 2016<br/> Check interval: 0 (&lt;none&gt;)<br/> Lifetime writes: 210 TB<br/> Reserved blocks uid: 0 (user root)<br/> Reserved blocks gid: 0 (group root)<br/> First inode: 11<br/> Inode size: 256<br/> Required extra isize: 28<br/> Desired extra isize: 28<br/> Journal inode: 8<br/> Default directory hash: half_md4<br/> Directory Hash Seed: c9240b0e-be05-4fb9-a604-69e902802452<br/> Journal backup: inode blocks<br/> MMP block number: 5638<br/> MMP update interval: 5<br/> User quota inode: 3<br/> Group quota inode: 4<br/> Journal features: journal_incompat_revoke<br/> Journal size: 400M<br/> Journal length: 102400<br/> Journal sequence: 0x01fe13aa<br/> Journal start: 33589</p> <p>If you are already running Lustre 2.8.x on the MDS/OSS then there isn't a huge amount to be done. You already have the <tt>dirdata</tt> feature (which has been available since 2.1) and <tt>flex_bg</tt>, which are the major performance gains compared to upgraded 1.8-formatted MDT filesystems. The above referenced issues were only relevant in case of a downgrade, but since you are already running 2.8, presumably without problems that would make you want to downgrade, I don't think they are relevant.</p> <p>The only other possible issue that would come up going forward is the size of the inodes on the MDT and OST. With Lustre 2.10+ we have bumped the default MDT inode size to 1024 bytes (from 512) and the default OST inode size to 512 (from 256) to facilitate usage of PFL in the future. If you are going to make PFL layouts the default on an ldiskfs MDT then you might consider to do a backup/restore (the inode size can only be changed at format time), but this is a non-issue for ZFS (which has dynamic dnode sizing as of 0.7.x).</p> <p>Yes are production system is running lustre 2.8 clients and lustre 2.8 servers. We have no plans with the current center wide file system to move to lustre 2.10. When the file system was created it was formated at a lustre 2.5 version. So we looking to see what needs to be abled to get to the 2.8 lustre support level. The big issue we have hit is when users create 20+ millions per directory which crashes are MDS server. I believe the large directory hash work in newer lustre versions fix this.</p> <p>James, we've never supported more than ~10M files per directory with ldiskfs (<a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#settinguplustresystem.tab2" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#settinguplustresystem.tab2</a>), except with DNE2 directories striped over multiple MDTs (each with &lt; 10M entries). The per-directory limit depends on the length of the filenames being used. Hitting this limit definitely shouldn't crash the MDS, which should be filed as a separate ticket with stack traces, etc.</p> <p>The <tt>large_dir</tt> feature hasn't been tested in production yet, as it has only landed in the development e2fsprogs release upstream, and the patches for e2fsprogs-wc need to be updated to include fixes made to those upstream patches. This definitely isn't something included as part of 2.8.</p> <p>So even tho ldiskfs has the code to support large_dir it is off by default and has never been really tested. You can't even set large_dir with the current ldiskfs version of e2fsprogs?</p> <p>Correct. Until recently, there was no e2fsck support for the <tt>large_dir</tt> feature, so it has not been safe to enable.</p> <p>Closing this issue, I don't think there is anything here to be done.</p> Related LU-7410 LU-8605 Development Rank 1|hzzh3b: Rank (Obsolete) 9223372036854775807