https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8068 Lustre <p>Unlanded patches exist in upstream ZFS to increase the dnode size which need to be evaluated for their impact (hopefully improvement) for Lustre metadata performance on ZFS MDTs:</p> <p><a href="https://github.com/zfsonlinux/zfs/pull/3542" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/3542</a></p> LU-8068

Large ZFS Dnode support

Improvement Minor Resolved Fixed Alex Zhuravlev Andreas Dilger llnl Fri, 21 Aug 2015 22:14:25 +0000 Mon, 27 Feb 2017 22:55:17 +0000 Thu, 2 Jun 2016 11:45:19 +0000 Lustre 2.9.0 0 19 <p>Any performance testing of this patch should be done using a Lustre MDT rather than just a local ZFS filesystem with a standard create/stat/unlink workload. Otherwise, the large dnodes will just slow down metadata performance due to increased IO, and the overhead of xattrs and external spill blocks used by Lustre will not be measured.</p> <p>It <em>may</em> be possible to use <tt>mdsrate &#45;&#45;create &#45;&#45;setxattr</tt> changes from <a href="https://jira.whamcloud.com/browse/LU-6483" title="Add xattrset to mdsrate" class="issue-link" data-issue-key="LU-6483"><del>LU-6483</del></a> (or equivalent) to test on a local ZFS filesystem, but this still needs an enhancement to allow storing smaller xattrs since &#45;&#45;setxattr currently only stores a 4000&#45;byte xattr. It needs to be enhanced to allow <tt>&#45;&#45;setxattr=&lt;size&gt;</tt> to store an xattr of a specific size, say 384 bytes for ZFS with 1024&#45;byte dnodes.</p> <p>Hi Jinshan,<br/> Can you have a look at this topic?<br/> Thanks.<br/> Joe</p> <p>It looks like this feature has some conflicts with storing XATTR as system attribute, which blocks further performance benchmark. I'm waiting for the author's response to move forward.</p> <p>I tried this patch with createmany on a directly mounted ZFS. it degrades create performance from ~29K/sec to ~20K/sec. I'm not sure how quickly this degradation can be addressed, but in general large dnode patch looks very important. to simulate it I tweaked the code to shrink LOVEA to just few bytes so that we fit bonus. and this brought creation rate from ~13K to ~20K in mds-survey.</p> <p>Alex, could you please post on the patch in GitHub so the LLNL folks can see. </p> <p>Also, it isn't clear what the difference is between your two tests. In the first case you wrote the create rate is <em>down</em> from 29k to 20k, is that for ZPL create rate? I don't expect this feature to help the non-Lustre case, since ZPL doesn't use SAs that can fit into the large dnode space, so it is just overhead. </p> <p>In the second case you wrote the create rate is <em>up</em> from 13k to 20k when you shrink the LOVEA, so presumably this is Lustre, but without the large dnode patch?</p> <p>What is the performance with Lustre with normal LOVEA size (1-4 stripes) + large dnodes? Presumably that would be 13k +/- some amount, not 29k +/- some amount?</p> <p>Also, my (vague) understanding of this patch is that it dynamically allocates space for the dnode, possibly using up space for dnode numbers following it? Does this fail if the dnode is not declared large enough for all future SAs during the initial allocation? IIRC, the osd-zfs code stores the layout and link xattrs to the dnode in a separate operation, which may make the large dnode patch ineffective. It may also have problems with multiple threads allocating dnodes from the same block in parallel, since it doesn't know at dnode allocation time how large the SA space Lustre eventually needs. Maybe my understanding of how this feature was implemented is wrong?</p> <p>Andreas, I've made a comment to github already, no reply so far. Hope Ned has that seen.</p> <p>so far I've tested large dnodes with ZPL only and noticed significant degradation, so I took a timeout hoping to see comments from Ned.<br/> I haven't tested Lustre with large dnodes.</p> <p>the patch allows to ask for dnode of specific size and I think we can do this given that we declare everything (including LOVEA of known size) ahead.<br/> we can easly track this in OSD.</p> <p>Rereading the large dnode patch, it seems that the caller can specify the dnode size on a per-dnode basis, so ideally we can add support for this to the osd-zfs code, but if not specified it will take the dataset property. Is 1KB large enough to hold the dnode + LOVEA + linkea + FID?</p> <p>Alex, your comment is on an old version of the patch, and not on the main pull request (<a href="https://github.com/zfsonlinux/zfs/pull/3542" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/3542</a>), so I don't think Ned will be looking there? Also, hopefully you are not using this old version of the patch (8f9fdb228), but rather the newest patch (ba39766)?</p> <p>yes, I was about to play with the code, but got confused by that performance issue. and yes, 1K should be more than enough: LinkEA would be 48+ bytes, LOVEA is something like 56+, then LMA and VBR (which I'd hope we can put into ZPL dnode, but in the worst case it's another 24+8 bytes). </p> <p>hmm, I was using old version.. let me try the new one. this will take some time - the patch doesn't apply to 0.6.5</p> <p>clean zfs/master:<br/> Created 1000000 in 29414ms in 1 threads - 33997/sec<br/> Created 1000000 in 20045ms in 2 threads - 49887/sec<br/> Created 1000000 in 19259ms in 4 threads - 51923/sec<br/> Created 1000000 in 17284ms in 8 threads - 57856/sec</p> <p>zfs/master + large dnodes:<br/> Created 1000000 in 40618ms in 1 threads - 24619/sec<br/> Created 1000000 in 28142ms in 2 threads - 35534/sec<br/> Created 1000000 in 25731ms in 4 threads - 38863/sec<br/> Created 1000000 in 25244ms in 8 threads - 39613/sec</p> <p>tried Lustre with that patch (on top of master ZFS):<br/> before:<br/> mdt 1 file 500000 dir 1 thr 1 create 21162.48 [ 18998.73, 22999.10] <br/> after:<br/> mdt 1 file 500000 dir 1 thr 1 create 18019.70 [ 15999.09, 19999.20] </p> <p>osd-zfs/ was modified to ask for 1K dnodes, verified with zdb:<br/> Object lvl iblk dblk dsize dnsize lsize %full type<br/> 10000 1 16K 512 0 1K 512 0.00 ZFS plain file</p> <p>notice zero dsize meaning no spill was allocated.</p> <p>Ned refreshed the patch to address that performance issue and now it's doing much better.<br/> first of all, now I'm able to complete some tests where I was getting OOM (because of huge memory consumption by 8K spill, I guess).<br/> now it makes sense to benchmark on a real storage as amount of IO with this patch is few times less:<br/> 1K vs (512byte dnode + 8K spill) per dnode OR 976MB vs 8300MB per 1M dnodes.</p> <p>The large dnode patch is blocked behind <a href="https://github.com/zfsonlinux/zfs/pull/4460" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/4460</a> which is the performance problem that Alex and Ned identified, but currently that patch is only a workaround and needs to be improved before landing. I've described in that ticket what seems to be a reasonable approach for making a production-ready solution, but to summarize:</p> <ul class="alternate" type="square"> <li>by default the dnode allocator should just use a counter that continues at the next file offset (as in the existing 4460 patch)</li> <li>if dnodes are being unlinked, a (per-cpu?) counter of unlinked dnodes and the minimum unlinked dnode number should be tracked (these values could be racy since it isn't critical that their values be 100% accurate)</li> <li>when the unlinked dnode counter exceeds some threshold (e.g. 4x number of inodes created in previous TXG, or 64x the number of dnodes that fit into a leaf block, or some tunable number of unlinked dnodes specified by userspace) then scanning should restart at the minimum unlinked dnode number instead of "0" to avoid scanning a large number of already-allocated dnode blocks</li> </ul> <p>Alex, in order to move the large dnode patch forward, could you or Nathaniel work on an updated 4460 patch so that we can get on with landing the large dnode patch.</p> <p>We're currently testing with the following patch to mitigate the performance impact of metadnode backfilling. It uses a naive heuristic (rescan after 4096 unlinks at most once per txg) but this is simple and probably achieves 99% of the performance to be gained here.</p> <p><a href="https://github.com/LLNL/zfs/commit/050b0e69" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/zfs/commit/050b0e69</a></p> <p>Ned Bass (bass6@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/20367" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20367</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8068" title="Large ZFS Dnode support" class="issue-link" data-issue-key="LU-8068"><del>LU-8068</del></a> osd-zfs: large dnode compatibility<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f0d8afaec213a7f471c3b22b9940de5c5cd192e3</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20367/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20367/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8068" title="Large ZFS Dnode support" class="issue-link" data-issue-key="LU-8068"><del>LU-8068</del></a> osd-zfs: large dnode support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9765c6174ef580fb4deef4e7faea6d5ed634b00f</p> <p>Landed for 2.9</p> Blocker LU-7895 Duplicate LU-8424 Related LU-6483 LU-8124 Development Rank 1|hzxl2f: Rank (Obsolete) 9223372036854775807