https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11246 Lustre <p>With the need to build e2fsprogs for ARM and Power8 I also attempted to build it for my general RHEL7 systems. I discovered a few things. First I think we can eliminate the extra spec files. If you install lsb_release for your distro it will build with the default spec file. A yum install redhat-release will do it. On my build machine I have libfuse-devel installed which broke the e2fsprogs. Even a ./configure --disable-fuse2fs didn't help. Lastly for my new target testbed the disk are 75TB in size so when I attempted to build a file system I saw the following error: </p> <p>mke2fs 1.44.3.wc1 (23-July-2018)</p> <p>mke2fs: Size of device (0x448000000 blocks) /dev/mapper/crius-ddn-l12 too big to be expressed in 32 bits using a blocksize of 4096.</p> <p>It looks like a -o 64 option will be needed.</p> Building and use latest modifies e2fsprogs for lustre. LU-11246

New lustre e2fsprogs 1.44 issues

Bug Critical Closed Duplicate Dongyang Li James A Simmons Tue, 14 Aug 2018 05:11:39 +0000 Wed, 17 Oct 2018 03:40:16 +0000 Wed, 17 Oct 2018 03:40:16 +0000 0 5 <blockquote> <p>With the need to build e2fsprogs for ARM and Power8 I also attempted to build it for my general RHEL7 systems. I discovered a few things. First I think we can eliminate the extra spec files. If you install <tt>lsb_release</tt> for your distro it will build with the default spec file. A <tt>yum install redhat-release</tt> will do it.</p></blockquote> <p>Firstly, I suspect that the <tt>yum install</tt> will not work for SLES, and it isn't clear if it would work for RHEL6? How does it know to get the right .spec file for the distro? Also, I suspect the vendor .spec file will not have the correct contact URLs and such, so it isn't clear to me how that would work. I'm not against reducing our maintenance overhead by eliminating patches, but you'd need to explain a bit more about what this is doing before I can understand whether it is actually something we want to change.</p> <blockquote> <p>On my build machine I have libfuse-devel installed which broke the e2fsprogs. Even a ./configure --disable-fuse2fs didn't help.</p></blockquote> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lidongyang" class="user-hover" rel="lidongyang">lidongyang</a> had a patch to add a configure check for <tt>libfuse-devel</tt> to skip this if it wasn't available. Maybe that is missing from the version that you were using? Or is the problem that this configure check is missing something and it still fails to build when <tt>libfuse-devel</tt> is not installed?</p> <blockquote> <p>Lastly for my new target testbed the disk are 75TB in size so when I attempted to build a file system I saw the following error: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mke2fs 1.44.3.wc1 (23-July-2018) mke2fs: Size of device (0x448000000 blocks) /dev/mapper/crius-ddn-l12 too big to be expressed in 32 bits using a blocksize of 4096. </pre> </div></div> <p>It looks like a -o 64 option will be needed.</p></blockquote> <p>I think we already add "<tt>-O 64bit</tt>" from <tt>mkfs.lustre</tt> by default for OSTs. Are you running <tt>mke2fs</tt> directly on the devices?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> enable_default_ext4_features() { <span class="code-comment">/* Enable large block addresses <span class="code-keyword">if</span> the LUN is over 2^32 blocks. */</span> <span class="code-keyword">if</span> ((mop-&gt;mo_device_kb / (L_BLOCK_SIZE &gt;&gt; 10) &gt; UINT32_MAX) &amp;&amp; is_e2fsprogs_feature_supp(<span class="code-quote">"-O 64bit"</span>) == 0) enable_64bit = 1; </pre> </div></div> <p>This has been the default even for MDTs since commit v2_10_58_0-123-geb65c3a patch <a href="https://review.whamcloud.com/31037" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31037</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10520" title="Cannot format large MDT with ldiskfs " class="issue-link" data-issue-key="LU-10520"><del>LU-10520</del></a> mkfs: enable extents for big MDT</tt>" because even though large MDTs were uncommon in the past, with DoM it is more likely that we will need large MDT filesystems. With a 75TB MDT holding 1B inodes, that is only about 64KB/file once the other filesystem overhead is taken into account.</p> <p>We do need the specs for the platforms to overcome the differences,</p> <p>e.g. several test case will always fail on a default SLES11 buildbox because it picks ext3 as the root filesystem. we need to apply extra patches and skip some tests for SLES only.</p> <p>When I first made the specs I put fuse-devel as a build dependency, but </p> <p>1. fuse2fs is not useful to us, it can not mount fs with ea_inode</p> <p>2. the build boxes from jenkins doesn't have fuse-devel by default</p> <p>so I've removed them from the specs, however the --disable-fuse2fs should go into the configure options in the specs. I've just fixed that in the git repo.</p> <p>Which repo did you push the fuse to?  I'm working off the master-lustre-test branch. </p> <p>@Andreas: I'm building the file system with mkfs.lustre using the tip of pre-2.12</p> <p>James, it's the same master-lustre-test branch, do a git pull or clone you should be able to see it.</p> <p>Last commit I see is "e2fsprogs: fix compile error and warnings for old gcc versions". Which commit is it?</p> <p>I was overwriting the commits, we want to keep the number of patches on top of e2fsprogs upstream to the minimum.</p> <p>One way to check is to look into the spec.in file, see if you have --disable-fuse2fs in the configure section.</p> <p>I managed to get my Power8 nodes running the lustre server code but using the latest e2fsprogs I'm seeing the following:</p> <p>[ 1049.122763] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.37.202.6@o2ib1 --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=250000 --reformat /dev/nvme0n1</p> <p>[ 1049.486023] LDISKFS-fs (nvme0n1): Couldn't mount because of unsupported optional features (400)</p> <p>Was something missed?</p> <p>400 is EXT4_FEATURE_INCOMPAT_EA_INODE, which should be there in ldiskfs</p> <p>what does the mkfs.lustre output say? like the options it passed to mke2fs?</p> <p>also what does dumpe2fs -h say?</p> <p>mkfs.lustre --mgsnode=172.30.224.8@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=md</p> <p>t.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=250000 -vvv --reformat /dev/nvme0n1</p> <p> </p> <p>   Permanent disk data:   </p> <p>Target:     lustre:MDT0000</p> <p>Index:      0</p> <p>Lustre FS:  lustre</p> <p>Mount type: ldiskfs</p> <p>Flags:      0x61</p> <p>              (MDT first_time update )</p> <p>Persistent mount opts: user_xattr,errors=remount-ro</p> <p>Parameters: mgsnode=172.30.224.8@tcp sys.timeout=20 mdt.identity_upcall=/usr/sbin/l_getidentity</p> <p> </p> <p>device size = 763097MB</p> <p>formatting backing filesystem ldiskfs on /dev/nvme0n1</p> <p>        target name   lustre:MDT0000</p> <p>        4k blocks     62500</p> <p>        options        -I 1024 -i 2560 -O dirdata,uninit_bg,^extents,quota,huge_file,ea_inode,flex_bg -E lazy_journal_init -F</p> <p>mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000  -I 1024 -i 2560 -O dirdata,uninit_bg,^extents,quota,huge_file,ea_inode,flex_bg -E lazy_journal_init -F /dev/nvme0n1 62500</p> <p>cmd: mke2fs -j -b 4096 -L lustre:MDT0000  -I 1024 -i 2560 -O dirdata,uninit_bg,^extents,quota,huge_file,ea_inode,flex_bg -E lazy_journal_init -F /dev/nvme0n1 62500</p> <p>mke2fs 1.44.3.wc1 (23-July-2018)</p> <p>/dev/nvme0n1 contains a ext4 file system labelled 'lustre:MDT0000'</p> <p>        created on Wed Sep  5 21:17:25 2018</p> <p>Discarding device blocks: done                            </p> <p>Creating filesystem with 62500 4k blocks and 100000 inodes</p> <p>Filesystem UUID: 509625e9-8319-4f61-87cf-bd60c2e2771f</p> <p>....</p> <p> </p> <p>Filesystem volume name:   lustre:MDT0000</p> <p>Last mounted on:          &lt;not available&gt;</p> <p>Filesystem UUID:          509625e9-8319-4f61-87cf-bd60c2e2771f</p> <p>Filesystem magic number:  0xEF53</p> <p>Filesystem revision #:    1 (dynamic)</p> <p>Filesystem features:      has_journal ext_attr resize_inode dir_index filetype flex_bg ea_inode dirdata sparse_super large_file huge_file uninit_bg quota</p> <p>Filesystem flags:         unsigned_directory_hash </p> <p>Default mount options:    user_xattr acl</p> <p>Filesystem state:         clean</p> <p>Errors behavior:          Continue</p> <p>Filesystem OS type:       Linux</p> <p> </p> <p>Now if I run </p> <p>mke2fs -j -b 4096 -L lustre:MDT0000  -I 1024 -i 2560 -O dirdata,uninit_bg,^extents,quota,huge_file,ea_inode,flex_bg -E lazy_journal_init -F /dev/nvme0n1 62500</p> <p>that works</p> <p>Also in the RHEL7 alt kernel I do see in ext4.h </p> <p>ext4.h:#define EXT4_FEATURE_INCOMPAT_<b>EA_INOD*E           0x0400 /</b> EA in inode */</p> <p>Sorry, it isn't clear in your next most recent comment what the problem is? It looks to me like the <tt>mkfs.lustre</tt> command was running OK. </p> <p>As for the feature flag definition in RHEL 7.3, that was just reserved for future use in that kernel, it doesn't mean the feature is actually supported by ext4. It would need to be in the <tt>EXT4_FEATURE_INCOMPAT_SUPP</tt> mask for the kernel allow it to mount.</p> <p>It seems like you don't have <tt>ext4-large-eas.patch</tt> applied. </p> <p>You are right. Looking at the RHEL7 alt kernel most of the ea_inode landed but not all. Only bits and pieces <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Okay I found the problem. For the RHLE7 alt kernel 4.14.0-49.6.1.el7a.ppc64le all the large eas work landed except for adding EXT4_FEATURE_INCOMPAT_EA_INODE to EXT4_FEATURE_INCOMPAT_SUPP. So it was a kernel bug. Have to talk to Oleg to see if the ARM kernel he is using has this bug as well.</p> <p> </p> <p>The only problem I have seen with e2fsprogs 1.44 is the LUN is to large issue. Let me see if I can reproduce and track down the problem.</p> <p>Janes. We've made the e2fsprogs-1.44.3-wc1 release - builds available under <a href="https://build.whamcloud.com/job/e2fsprogs-master/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/e2fsprogs-master/</a> but not pushed to the download site while they undergo final testing. </p> <p>Could you please explain what the "LUN atop large" issue is? It isn't referenced anywhere else in this ticket. </p> <p>If you aren't having any more problems I'd like to close this, as it is mostly a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11440" title="Make e2fsprogs-1.44.3-wc1 release" class="issue-link" data-issue-key="LU-11440"><del>LU-11440</del></a>. </p> <p>Still need to look into the to large LUN error.</p> Blocker LU-11268 Related LU-1732 LU-6387 LU-11200 LU-11215 LU-10997 LU-11440 Development Rank 1|i000p3: Rank (Obsolete) 9223372036854775807 Severity