[#LU-419] Running 1.8.6WC the mkfs.lustre failed to format a 22.94TB LUN, Size of device too big to be expressed in 32 bits using a block size of 4096

[LU-419] Running 1.8.6WC the mkfs.lustre failed to format a 22.94TB LUN, Size of device too big to be expressed in 32 bits using a block size of 4096 Created: 15/Jun/11 Updated: 26/Oct/11 Resolved: 11/Aug/11
Status:	Closed
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.6
Fix Version/s:	Lustre 2.1.0, Lustre 1.8.7

Type:

Bug

Priority:

Major

Reporter:

John Salinas (Inactive)

Assignee:

Andreas Dilger

Resolution:

Fixed

Votes:

Labels:

None

Environment:

Centos/Rhel 5.5 Linux servers running 1.8.6WC

Story Points:

Severity:

Rank (Obsolete):

4974

Description

rpm -qa |grep -i lustre
kernel-headers-2.6.18-238.12.1.el5_lustre
lustre-1.8.6-wc1_2.6.18_238.12.1.el5_lustre
ddn-sfa-ofed-1.5.3.1-2.6.18_238.12.1.el5_lustre
lustre-modules-1.8.6-wc1_2.6.18_238.12.1.el5_lustre
kernel-2.6.18-238.12.1.el5_lustre
kernel-devel-2.6.18-238.12.1.el5_lustre
kernel-ib-1.5.3.1-2.6.18_238.12.1.el5_lustre
kernel-ib-devel-1.5.3.1-2.6.18_238.12.1.el5_lustre
ddn-lustre-tools-0.4-2011.06.08.123659
lustre-ldiskfs-3.1.50-wc1_2.6.18_238.12.1.el5_lustre

uname -a
Linux oss0 2.6.18-238.12.1.el5_lustre #1 SMP Fri Jun 10 19:09:49 MDT 2011 x86_64 x86_64 x86_64 GNU/Linux

fdisk -l /dev/ddn/ost_testfs_54

Disk /dev/ddn/ost_testfs_54: 23493.4 GB, 23493471109120 bytes
255 heads, 63 sectors/track, 2856251 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/ddn/ost_testfs_54 doesn't contain a valid partition table

mkfs.lustre --mgsnode=127.0.0.2@tcp --failnode=127.0.0.2@tcp --fs
name=testfs --ost --mkfsoptions=-m1 --index=9 /dev/ddn/ost_testfs_54

Permanent disk data:
Target: testfs-OST0009
Index: 9
Lustre FS: testfs
Mount type: ldiskfs
Flags: 0x62
(OST first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=127.0.0.2@tcp failover.node=127.0.0.2@tcp

checking for existing Lustre data: not found
device size = 22405120MB
2 6 18
formatting backing filesystem ldiskfs on /dev/ddn/ost_testfs_54
target name testfs-OST0009
4k blocks 5735710720
options -m1 -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups,mmp -F
mkfs_cmd = mke2fs -j -b 4096 -L testfs-OST0009 -m1 -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups,mmp -F /dev/ddn/ost_testfs_54 5735710720

Warning: the fs_type huge is not defined in mke2fs.conf

mke2fs: Size of device (0x155e00000 blocks) /dev/ddn/ost_testfs_54 too big to be expressed
in 32 bits using a blocksize of 4096.

mkfs.lustre FATAL: Unable to build fs /dev/ddn/ost_testfs_54 (256)

mkfs.lustre FATAL: mkfs failed 256

This also failed yesterday when we built out of b1_8:

rpm -qa |grep -i lustre
lustre-debuginfo-1.8.5.56-2.6.18_238.9.1.el5_lustre.20110528080747
kernel-ib-devel-1.5.3.1-2.6.18_238.9.1.el5_lustre.20110525141730
kernel-headers-2.6.18-238.9.1.el5_lustre.20110525141730
lustre-ldiskfs-debuginfo-3.1.50-2.6.18_238.9.1.el5_lustre.20110528080747
lustre-modules-1.8.5.56-2.6.18_238.9.1.el5_lustre.20110528080747
lustre-1.8.5.56-2.6.18_238.9.1.el5_lustre.20110528080747
kernel-devel-2.6.18-238.9.1.el5_lustre.20110525141730
kernel-2.6.18-238.9.1.el5_lustre.20110525141730
kernel-debuginfo-common-2.6.18-238.9.1.el5_lustre.20110525141730
kernel-debuginfo-2.6.18-238.9.1.el5_lustre.20110525141730
ddn-lustre-tools-0.4-2011.04.01.153313
lustre-ldiskfs-3.1.50-2.6.18_238.9.1.el5_lustre.20110528080747
ddn-sfa-ofed-1.5.3.1-2.6.18_238.9.1.el5_lustre.20110525141730
kernel-ib-1.5.3.1-2.6.18_238.9.1.el5_lustre.20110525141730

mkfs.lustre --mgsnode=127.0.0.2@tcp --failnode=127.0.0.2@tcp --fsname=testfs --ost --mkfsoptions=-m1 -i 131072 --index=54 /dev/ddn/ost_testfs_54

Permanent disk data:
Target: testfs-OST0036
Index: 54
Lustre FS: testfs
Mount type: ldiskfs
Flags: 0x62
(OST first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=127.0.0.2@tcp failover.node=127.0.0.2@tcp

checking for existing Lustre data: not found
device size = 22405120MB
2 6 18
formatting backing filesystem ldiskfs on /dev/ddn/ost_testfs_54
target name testfs-OST0036
4k blocks 5735710720
options -m1 -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups,mmp -F
mkfs_cmd = mke2fs -j -b 4096 -L testfs-OST0036 -m1 -J size=400 -i 16384 -I 256 -q -O dir_index,extents,uninit_groups,mmp -F /dev/ddn/ost_testfs_54 5735710720

Warning: the fs_type huge is not defined in mke2fs.conf

mke2fs: Size of device (0x155e00000 blocks) /dev/ddn/ost_testfs_54 too big to be expressed
in 32 bits using a blocksize of 4096.

mkfs.lustre FATAL: Unable to build fs /dev/ddn/ost_testfs_54 (256)

mkfs.lustre FATAL: mkfs failed 256

Our simple attempt to increase the block size to 8192 not work either:

Warning: blocksize 8192 not usable on most systems.
Warning: 8192-byte blocks too big for system (max 4096), forced to continue

Warning: the fs_type huge is not defined in mke2fs.conf

mke2fs: Size of device (0x155e00000 blocks) /dev/ddn/ost_testfs_54 too big to be expressed
in 32 bits using a blocksize of 8192.

Comments

Comment by Peter Jones [ 15/Jun/11 ]

Andreas

Can you please comment?

Thanks

Peter

Comment by Andreas Dilger [ 15/Jun/11 ]

What version of e2fsprogs is being used? You should use the latest build of e2fsprogs, available from:

http://downloads.whamcloud.com/public/e2fsprogs/latest/

Currently that is e2fsprogs-1.41.90-wc2, which is the only version that supports filesystems over 16TB.

Comment by John Salinas (Inactive) [ 15/Jun/11 ]

rpm -qa |grep -i e2fsprogs
e2fsprogs-1.41.90.wc2-0redhat

Comment by John Salinas (Inactive) [ 15/Jun/11 ]

/sbin/e2fsck -V
e2fsck 1.41.90.wc2 (14-May-2011)
Using EXT2FS Library version 1.41.90.wc2, 14-May-2011

Comment by Andreas Dilger [ 15/Jun/11 ]

Also, in order to make your filesystem MUCH faster to format and run e2fsck, you should enable the "flex_bg" feature and other ext4-specific options using "-t ext4", and reduce the number of inodes created on the filesystem (average object size of 256kB instead of the current default of 16kB):

mkfs.lustre --mkfsoptions="-t ext4 -i 262144" /dev/ddn/ost_testfs_54

Comment by John Salinas (Inactive) [ 15/Jun/11 ]

Isn't 1.8.6WC ext4 only? Why would this need to an added option instead of default? I seem to recall as patch that was going to make this default did that not make it into 1.8.6WC RC2 or was I mistaken?

Comment by Andreas Dilger [ 15/Jun/11 ]

There was a change in ~~LU-255~~ that added this as the default for "master" but it was not landed for 1.8.6.

I'm just working on a patch for 1.8.6.

Comment by John Salinas (Inactive) [ 15/Jun/11 ]

I would definitely vote for an RC3 with this patch.

Comment by Peter Jones [ 15/Jun/11 ]

John

Just to inject a note of caution here - I am not sure whether it would be appropriate to land this patch to 1.8.x. Andreas is preparing a patch "just in case" but for now I would plan to specify the options in order to complete your testing. For the master codeline we are only supporting ext4, but even though we have chosen to focus on ext4 only for this initial 1.8.x release, we may need to keep the door open so we can still provide ext3 options for future 1.8.x releases. We can discuss this as we review the RC2 testing results but I think that all round we would prefer to avoid an RC3 so as not to jeopardize the timeline. The most important thing is to get your testing progressing again. Do you have enough information to accomplish that?

Regards

Peter

Comment by Peter Jones [ 15/Jun/11 ]

John

Can you please provide a status update? Are you now able to test or, if not, what is still blocking you?

Please advise

Peter

Comment by John Salinas (Inactive) [ 15/Jun/11 ]

We appear to be able to make the file system on the device. Other issues have prevented the device from being tested and allow us to confirm that what was made does indeed work - these are just setup issues that we are working through.

Comment by Peter Jones [ 15/Jun/11 ]

ok, thanks John. Do keep us updated with your progress and let us know if there is anything that we can do to help.

Comment by Peter Jones [ 16/Jun/11 ]

John

Just checking in to see how things are going. Where do things stand atm?

Peter

Comment by Andreas Dilger [ 17/Jun/11 ]

Patch to automatically enable formatting > 16TB LUNs and enabling other ext4 features (especially flex_bg) by default is in http://review.whamcloud.com/#change,958.