[LU-11181] df -i and lfs df -i show incorrect inode count Created: 26/Jul/18 Updated: 02/Feb/21 Resolved: 07/Sep/18 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.10.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Mahmoud Hanafi | Assignee: | Andreas Dilger |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Severity: | 2 | ||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
lfs df -i and df -i doen't report correct inode counts. On a newly created filesystem we have 2 MDTs. Here is the info on the MDTs. nbp10-srv1 ~ # tune2fs -l /dev/mapper/nbp10_1-MDT0 tune2fs 1.42.13.wc6 (05-Feb-2017) device /dev/dm-1 mounted by lustre Filesystem volume name: nbp10-MDT0000 Last mounted on: / Filesystem UUID: bc01a76f-c04d-4f35-966b-e3e655a12528 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4241293312 Block count: 3106406400 Reserved block count: 0 Free blocks: 2044739588 Free inodes: 4241293299 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 387 Blocks per group: 24000 Fragments per group: 24000 Inodes per group: 32768 Inode blocks per group: 8192 Flex block group size: 16 Filesystem created: Wed Jul 25 21:43:46 2018 Last mount time: Wed Jul 25 21:46:39 2018 Last write time: Wed Jul 25 21:46:45 2018 Mount count: 3 Maximum mount count: -1 Last checked: Wed Jul 25 21:43:46 2018 Check interval: 0 (<none>) Lifetime writes: 196 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 1024 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 9b03f637-f156-4f57-aeff-e6872c65eb67 Journal backup: inode blocks MMP block number: 17822 MMP update interval: 5 User quota inode: 3 Group quota inode: 4 nbp10-srv3 ~ # tune2fs -l /dev/mapper/nbp10_2-MDT1 tune2fs 1.42.13.wc6 (05-Feb-2017) device /dev/dm-0 mounted by lustre Filesystem volume name: nbp10-MDT0001 Last mounted on: / Filesystem UUID: eb11a0e3-772a-4e83-9177-08d41210c8b7 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4241293312 Block count: 3106406400 Reserved block count: 0 Free blocks: 2044739588 Free inodes: 4241293299 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 387 Blocks per group: 24000 Fragments per group: 24000 Inodes per group: 32768 Inode blocks per group: 8192 Flex block group size: 16 Filesystem created: Wed Jul 25 21:43:48 2018 Last mount time: Wed Jul 25 21:46:39 2018 Last write time: Wed Jul 25 21:46:45 2018 Mount count: 3 Maximum mount count: -1 Last checked: Wed Jul 25 21:43:48 2018 Check interval: 0 (<none>) Lifetime writes: 196 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 1024 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 76bf0136-01da-4bd1-b383-21d9ec90ee17 Journal backup: inode blocks MMP block number: 17822 MMP update interval: 5 User quota inode: 3 Group quota inode: 4 $ lfs df -i |grep nobackupp10 |grep -e summary -e MDT $ df -i /nobackupp10 |
| Comments |
| Comment by Gerrit Updater [ 26/Jul/18 ] |
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32887 |
| Comment by Andreas Dilger [ 26/Jul/18 ] |
|
You excluded the lfs df -i output from the OSTs, but I suspect the reason for this difference is because lfs df -i and df -i (via statfs()) limit the number of inodes shown to available objects on the OSTs (also taking the default stripe count into account) if it is lower than the number of inodes on the MDT(s). This is because even if you have 8B inodes on the MDTs, you also need to allocate at least one OST object for each MDT object in order to store data in that file. If you plan on adding new OSTs in the future, or creating zero-length files (e.g. where only the filename/xattrs are important) then this is fine. Otherwise, your MDTs are over-provisioned and will not be able to use the full inode count. If this is a new filesystem, I would recommend to reduce the MDT inodes count to better match the OST inode count. Depending on your file usage patterns, and if the MDT is stored on flash (which I strongly recommend for all new deployments) you may want to increase the inode ratio (space per inode) from the default 2.5KB/inode to something significantly larger, say 64KB/inode to allow future use by the Data-on-MDT feature that is available in Lustre 2.11 and later. Using mkfs.lustre ... --mkfsoptions="-i 65536" to allocate one inode per 64KB of space would reduce the number of inodes on the two MDTs from 8B to 380M, but it would allow significant performance improvements in the future (see Small File IO Performance in Lustre and Early Performance Investigation of Data on Metadata for more details). I've also pushed an update to the Lustre Manual to improve the section on formatting ldiskfs MDTs to explain this better. |
| Comment by Gerrit Updater [ 07/Aug/18 ] |
|
Joseph Gmitter (jgmitter@whamcloud.com) merged in patch https://review.whamcloud.com/32887/ |