https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5391 Lustre <p>For example, on a MDS an oi.xx directory:<br/> Object lvl iblk dblk dsize lsize %full type<br/> 44 4 4K 4K 1.06M 4.04M 99.71 ZFS directory</p> <p>The code in __osd_zap_create():<br/> oid = zap_create_flags(uos-&gt;os, 0, flags | ZAP_FLAG_HASH64, DMU_OT_DIRECTORY_CONTENTS, 12, 12, DMU_OT_SA, DN_MAX_BONUSLEN, tx);</p> <p>This seemed inefficient. In fact the default leaf block size for fat ZAP is 16K:<br/> int fzap_default_block_shift = 14; /* 16k blocksize */</p> <p>I changed the block sizes to 16K indirect and 128K leaf, and saw a 10% increase in mds-survey creation rates.</p> LU-5391

osd-zfs: ZAP objects use 4K blocks for both indirect and leaf blocks

Bug Major Resolved Fixed Isaac Huang Isaac Huang prz zfs Tue, 22 Jul 2014 05:29:12 +0000 Fri, 26 Sep 2014 04:20:49 +0000 Tue, 26 Aug 2014 15:24:13 +0000 Lustre 2.7.0 Lustre 2.7.0 0 6 <p>Brian or Alex, can you comment?</p> <p>4K seem to be small, of course, but I don't think 128K is good either. this might be OK for a relatively small directories, but when a directory is big, then evenly distributed load will be touching many different blocks leading to very low write density - we'll have to make 128K I/O for very few entries to modify. I'd suggest to stay with default 16K.</p> <p>I agree that 128K was too aggressive. I changed to 16K/16K and still got 7.4% and 14% increases (over current 4K/4K) for mds-survey creation and destroy rates respectively.</p> <p>I'd suggest to increase indirect block to 16K, which is the default indirect block size used by ZPL directories (e.g. / on a MDT), to reduce levels of indirection, and to increase leaf block to 16K, which matches fzap_default_block_shift.</p> <p>Patch pushed to <a href="http://review.whamcloud.com/#/c/11182/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11182/</a></p> <p>This is going to be a trade off between memory usage, bandwidth, and IO/s. I think adopting the ZPL defaults of 16k strikes a good balance but I'd be careful about drawing any performance conclusions. This may help create performance but it will hurt other workloads.</p> <p>For example, I'd expect that increasing the OI leaf block size on the MDS would improve performance as long as the entire OI can be cached (small filesystems). But once the OI size is significantly larger than memory (large filesystems) I see two downsides. 1) pulling in a larger block takes slightly longer, and 2) because the FIDs are hashed uniformly over the leaves this effectively reduces by 4x the cached working set size for a given set of FIDs.</p> <p>Brian, thanks for the comment:<br/> 1. The single-node mds-survey benchmark was way simplistic and it was not intended as any performance conclusion.<br/> 2. When you said "effectively reduces by 4x the cached working set size", did you mean: the FIDs in the working set are so sparse that each leaf block holds only one in the set, so for a fixed cache size cached working set is reduced by 4?</p> <p>&gt; the FIDs in the working set are so sparse that each leaf block holds only one...</p> <p>Yes, exactly. From everything I've seen the FIDs are distributed very uniformly over the leaves. This is a good thing and a bad thing. So let's say you have a set of 1024 files all of which are being frequently accessed so the OI leaf blocks all end up on the ARCs MFU. If the OI FatZAP has enough total entries my expectation would be that each entry would hash to a different leaf. So with 4K leaf blocks you'll consume roughly 4M of memory. With 16k leak blocks you're looking at 16M for the same workloads. This may still be a reasonable trade-off to make, but it's something which should be considered.</p> <p>Landed for 2.7</p> Development Rank 1|hzwryv: Rank (Obsolete) 15006 Severity