https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9176 Lustre <p>LFS configuration with two DNE2 MDTs, one MDT per MDS pair. LFS shows support for 215M files.</p> <p>MDT0000: 3.3TB zpool<br/> MDT0001: 3.7TB zpool</p> <ol> <li>lfs df<br/> UUID 1K-blocks Used Available Use% Mounted on<br/> data-0-MDT0000_UUID 3374354432 9472 3374342912 0% /mnt/data-0<span class="error">&#91;MDT:0&#93;</span><br/> data-0-MDT0001_UUID 3749283200 9344 3749271808 0% /mnt/data-0<span class="error">&#91;MDT:1&#93;</span></li> </ol> <ol> <li>lfs df -i<br/> UUID Inodes IUsed IFree IUse% Mounted on<br/> data-0-MDT0000_UUID 102359233 259 102358974 0% /mnt/data-0<span class="error">&#91;MDT:0&#93;</span><br/> data-0-MDT0001_UUID 113503655 248 113503407 0% /mnt/data-0<span class="error">&#91;MDT:1&#93;</span></li> </ol> <p>A single 3.3TB MDT should result in ~800M files. A 3.3TB and 3.7TB MDT I should be seeing a lot more than 215M files.</p> <p>What am I missing?</p> CentOS 7.3, EE 3.1.0.3, ZFS 0.6.5.7 LU-9176

ZFS MDT sizing. 7TB of MDT shows LFS support for 215M files?

Bug Major Resolved Fixed Joseph Gmitter Jeff Johnson Wed, 1 Mar 2017 23:13:30 +0000 Sat, 18 Mar 2017 13:39:40 +0000 Thu, 16 Mar 2017 22:03:12 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.10.0 0 8 <p>I seem to recall this being a ZFS "feature" (peculiarity). </p> <p>MDT0000: 3455338938368 bytes<br/> MDT0001: 3839265996800 bytes<br/> Total: 7294604935168 bytes / 215862888 = approx 33793 bytes per inode</p> <p>Seems a bit high.</p> <p>Jeff, the number of available inodes in the ZFS filesystem is an estimate based on the average space used per file so far, since there isn't a fixed inode table. If the MDTs are mostly empty (only ~250 inodes allocated), which are mostly directories (e.g. 128 <tt>OI.n</tt> ZAPs and 32 directories per <tt>O/{0,1,10,200000003</tt>} created at mount time), then the numbers will be skewed to a very conservative inode count estimate.</p> <p>If you create a few thousand regular files this should get closer to the expected 4KB/inode for ZFS, as described in <a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#settinguplustresystem.tab2" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#settinguplustresystem.tab2</a> . I'm updating the "Determining MDT Space Requirements" section of the manual to contain more information about ZFS.</p> <p>Presumably you haven't set a <tt>recordsize=N</tt> option on the MDT in this case?</p> <p>No, I left it at default after learning the hard way in the past about setting the recordsize to too low of a number.</p> <p>I updated the MDT sizing manual in <a href="https://review.whamcloud.com/25713" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25713</a> , and a formatted version of this section is available at <a href="https://build.hpdd.intel.com/job/lustre-manual-reviews/754/artifact/lustre_manual.xhtml#dbdoclet.space_requirements" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual-reviews/754/artifact/lustre_manual.xhtml#dbdoclet.space_requirements</a> if you are interested.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25743" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25743</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9176" title="ZFS MDT sizing. 7TB of MDT shows LFS support for 215M files?" class="issue-link" data-issue-key="LU-9176"><del>LU-9176</del></a> osd-zfs: improve statfs estimate for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4df3840ae497820904caee074d9e30e649da8e61</p> <p>Jeff, with the attached patch to fix up the inode estimation (increasing the number of "synthetic average sized inodes" added to the estimate when the filesystem is nearly empty, and fixing the calculation a bit) MDT0000 would report about 568M inodes (synthetic average inode size = 6KB rather than 33KB previously) when newly formatted, which is much closer to the expected 843M 4KB inodes for the 3.3TB filesystem. With 40k real inodes in the filesystem, the synthetic average would only contribute 10% to the average, and at 1M inodes it would contribute only 0.4% to the average size.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25743/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25743/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9176" title="ZFS MDT sizing. 7TB of MDT shows LFS support for 215M files?" class="issue-link" data-issue-key="LU-9176"><del>LU-9176</del></a> osd-zfs: improve statfs estimate for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5e2b17964fd880a9763ba66151aab6091fb4813c</p> <p>Landed for 2.10</p> Related Development Rank 1|hzz5lj: Rank (Obsolete) 9223372036854775807 Severity Site Affected: GD