https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11181 Lustre <p>lfs df -i and df -i doen't report correct inode counts.</p> <p>On a newly created filesystem we have 2 MDTs. Here is the info on the MDTs.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp10-srv1 ~ # tune2fs -l /dev/mapper/nbp10_1-MDT0 tune2fs 1.42.13.wc6 (05-Feb-2017) device /dev/dm-1 mounted by lustre Filesystem volume name: nbp10-MDT0000 Last mounted on: / Filesystem UUID: bc01a76f-c04d-4f35-966b-e3e655a12528 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4241293312 Block count: 3106406400 Reserved block count: 0 Free blocks: 2044739588 Free inodes: 4241293299 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 387 Blocks per group: 24000 Fragments per group: 24000 Inodes per group: 32768 Inode blocks per group: 8192 Flex block group size: 16 Filesystem created: Wed Jul 25 21:43:46 2018 Last mount time: Wed Jul 25 21:46:39 2018 Last write time: Wed Jul 25 21:46:45 2018 Mount count: 3 Maximum mount count: -1 Last checked: Wed Jul 25 21:43:46 2018 Check interval: 0 (&lt;none&gt;) Lifetime writes: 196 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 1024 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 9b03f637-f156-4f57-aeff-e6872c65eb67 Journal backup: inode blocks MMP block number: 17822 MMP update interval: 5 User quota inode: 3 Group quota inode: 4 nbp10-srv3 ~ # tune2fs -l /dev/mapper/nbp10_2-MDT1 tune2fs 1.42.13.wc6 (05-Feb-2017) device /dev/dm-0 mounted by lustre Filesystem volume name: nbp10-MDT0001 Last mounted on: / Filesystem UUID: eb11a0e3-772a-4e83-9177-08d41210c8b7 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4241293312 Block count: 3106406400 Reserved block count: 0 Free blocks: 2044739588 Free inodes: 4241293299 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 387 Blocks per group: 24000 Fragments per group: 24000 Inodes per group: 32768 Inode blocks per group: 8192 Flex block group size: 16 Filesystem created: Wed Jul 25 21:43:48 2018 Last mount time: Wed Jul 25 21:46:39 2018 Last write time: Wed Jul 25 21:46:45 2018 Mount count: 3 Maximum mount count: -1 Last checked: Wed Jul 25 21:43:48 2018 Check interval: 0 (&lt;none&gt;) Lifetime writes: 196 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 1024 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 76bf0136-01da-4bd1-b383-21d9ec90ee17 Journal backup: inode blocks MMP block number: 17822 MMP update interval: 5 User quota inode: 3 Group quota inode: 4 </pre> </div></div> <p>$ lfs df -i |grep nobackupp10 |grep -e summary -e MDT<br/> nbp10-MDT0000_UUID 4241293312 351 4241292961 0% /nobackupp10<span class="error">&#91;MDT:0&#93;</span><br/> nbp10-MDT0001_UUID 4241293312 337 4241292975 0% /nobackupp10<span class="error">&#91;MDT:1&#93;</span><br/> filesystem_summary: 429501508 688 429500820 0% /nobackupp10</p> <p>$ df -i /nobackupp10<br/> Filesystem Inodes IUsed IFree IUse% Mounted on<br/> 10.151.26.159@o2ib:10.151.26.160@o2ib:/nbp10 429501508 688 429500820 1% /nobackupp10</p> LU-11181

df -i and lfs df -i show incorrect inode count

Bug Major Resolved Fixed Andreas Dilger Mahmoud Hanafi Thu, 26 Jul 2018 17:25:53 +0000 Tue, 2 Feb 2021 19:24:37 +0000 Fri, 7 Sep 2018 20:21:24 +0000 Lustre 2.10.3 0 5 <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32887" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32887</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11181" title="df -i and lfs df -i show incorrect inode count" class="issue-link" data-issue-key="LU-11181"><del>LU-11181</del></a> mdt: include DoM in inode ratio discussion<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b5150a017b75592f93dee512ae27be4a97165dcf</p> <p>You excluded the <tt>lfs df -i</tt> output from the OSTs, but I suspect the reason for this difference is because <tt>lfs df -i</tt> and <tt>df -i</tt> (via <tt>statfs()</tt>) limit the number of inodes shown to <b>available</b> objects on the OSTs (also taking the default stripe count into account) if it is lower than the number of inodes on the MDT(s). This is because even if you have 8B inodes on the MDTs, you also need to allocate at least one OST object for each MDT object in order to store data in that file.</p> <p>If you plan on adding new OSTs in the future, or creating zero-length files (e.g. where only the filename/xattrs are important) then this is fine. Otherwise, your MDTs are over-provisioned and will not be able to use the full inode count. </p> <p>If this is a new filesystem, I would recommend to reduce the MDT inodes count to better match the OST inode count. Depending on your file usage patterns, and if the MDT is stored on flash (which I strongly recommend for all new deployments) you may want to increase the <em>inode ratio</em> (space per inode) from the default 2.5KB/inode to something significantly larger, say 64KB/inode to allow future use by the Data-on-MDT feature that is available in Lustre 2.11 and later. Using <tt>mkfs.lustre ... --mkfsoptions="-i 65536"</tt> to allocate one inode per 64KB of space would reduce the number of inodes on the two MDTs from 8B to 380M, but it would allow significant performance improvements in the future (see <a href="http://wiki.lustre.org/images/c/c5/LUG2018-Small_File_IO_Perf_DataOnMDT-Gmitter.pdf" class="external-link" target="_blank" rel="nofollow noopener">Small File IO Performance in Lustre</a> and <a href="http://wiki.lustre.org/images/9/94/LUG2018-Lustre_Data_on_MDT_An_Early_Look-Leers.pdf" class="external-link" target="_blank" rel="nofollow noopener">Early Performance Investigation of Data on Metadata</a> for more details).</p> <p>I've also pushed an update to the Lustre Manual to improve the section on formatting ldiskfs MDTs to explain this better.</p> <p>Joseph Gmitter (jgmitter@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32887/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32887/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11181" title="df -i and lfs df -i show incorrect inode count" class="issue-link" data-issue-key="LU-11181"><del>LU-11181</del></a> mdt: include DoM in inode ratio discussion<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5fd65770942c45e10f88d8d414ce799937931392</p> Related LUDOC-458 Development Rank 1|hzzzwf: Rank (Obsolete) 9223372036854775807 Severity