Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-7898] remove unnecessary declarations from osd-zfs https://jira.whamcloud.com/browse/LU-7898 Lustre <p>in contrast with ldiskfs, declarations in ZFS can be very expensive (even more expensive than execution itself). there are number of declarations in osd-zfs which can be removed safely. </p> LU-7898

remove unnecessary declarations from osd-zfs

Improvement Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Tue, 22 Mar 2016 14:25:34 +0000 Thu, 2 Feb 2017 16:10:06 +0000 Sat, 10 Sep 2016 04:22:21 +0000 Lustre 2.9.0 0 5 <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19101" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19101</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a25217845bc24d21648e8dffc6863a9ccb1813f8</p> <p>I've took to-write and to-overwrite estimations for createmany -m and createmany -o, in Kbs. the sum of these estimations then used to reserve space in TXG and ARC.</p> <p>clean master:<br/> createmany -m:<br/> 40 thx – 2w=16160 2ov=3072<br/> createmany -o:<br/> 73 thx – 2w=27536 2ov=6048</p> <p><a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a>:<br/> createmany -m:<br/> 12 thx – 2w=4592 2ov=1616<br/> createmany -o:<br/> 25 thx – 2w=8800 2ov=3472</p> <p><a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> -dnode accounting:<br/> createmany -m:<br/> 8 thx – 2w=2800 2ov=1168<br/> createmany -o:<br/> 17 thx – 2w=5216 2ov=2576</p> <p>this is how credits are distributed for a single-stripe file creation with this patch, in Kbs:</p> <p>8 – 2w=3568 + 2ov=1040 = 4608<br/> 0: 2w=112 2ov=0 = 112 // dmu_object_alloc()<br/> 6: 2w=896 2ov=112 = 1008 // dmu_tx_hold_zap(tx, sa->sa_layout_attr_obj<br/> 8: 2w=896 2ov=112 = 1008 // OI<br/> 136: 2w=384 2ov=112 = 496 // iusr zap<br/> 137: 2w=384 2ov=112 = 496 // igrp zap<br/> 200: 2w=0 2ov=240 = 240 // lovobjids<br/> 169: 2w=896 2ov=112 = 1008 // dir insert<br/> 152: 2w=0 2ov=240 = 240 // last_rcvd</p> <p>it's clear that ZAP's related credits consume the majority of the whole reservation.<br/> first of all, it's currently not very optimal because calculated using SPA_OLD_MAXBLOCKSIZE which is 128K:</p> <p> *towrite += (3 + (add ? 4 : 0)) * SPA_OLD_MAXBLOCKSIZE;</p> <p>while by default ZAP's blocksize: int fzap_default_block_shift = 14; /* 16k blocksize */<br/> so, if we change the code to use actual blocksize, then we can shrink that 896K to 112K.<br/> with dnode allocation move into ZFS, that would lead to ~1M of credits for a single-stripe file creation.<br/> current master reserves ~33,5MB.</p> <p>unlink is still rather heavy due to llog:<br/> 19 – 2w=5616 + 2ov=2656 = 8272<br/> 169: 2w=384 2ov=112 = 496 // dir delete<br/> 207: 2w=0 2ov=112 = 112 // first ref_del on child?<br/> 8: 2w=384 2ov=112 = 496 // OI<br/> 136: 2w=384 2ov=112 = 496 // quota usr zap <br/> 137: 2w=384 2ov=112 = 496 // quota grp zap<br/> 207: 2w=0 2ov=224 = 224 // dmu_tx_hold_free(child)<br/> 204: 2w=0 2ov=240 = 240 // write to llog header<br/> 204: 2w=256 2ov=112 = 368 // append to llog<br/> 142: 2w=0 2ov=240 = 240 // add new llog to catalog<br/> 0: 2w=112 2ov=0 = 112 L1 // new llog object create<br/> 6: 2w=896 2ov=112 = 1008 // dmu_tx_hold_zap(tx, sa->sa_layout_attr_obj<br/> 8: 2w=896 2ov=112 = 1008 // OI for new llog object<br/> 136: 2w=384 2ov=112 = 496 // quota usr zap<br/> 137: 2w=384 2ov=112 = 496 // quota grp zap<br/> 203: 2w=0 2ov=240 = 240 // init new llog<br/> 203: 2w=256 2ov=112 = 368 // write to new llog<br/> 173: 2w=896 2ov=112 = 1008 // PENDING<br/> 207: 2w=0 2ov=128 = 128 // LinkEA<br/> 152: 2w=0 2ov=240 = 240 // last_rcvd</p> <p>with dnode accounting moved to ZFS and zap blocksize=16k that would be 2774K.</p> <p>llog is still very expensive.. with the patch a single record needs 4576K reserved:<br/> reservations per llog record - 4576K:<br/> 204: 2w=0 2ov=240 = 240 // write to llog header<br/> 204: 2w=256 2ov=112 = 368 // append to llog<br/> 142: 2w=0 2ov=240 = 240 // add new llog to catalog<br/> 0: 2w=112 2ov=0 = 112 // new llog object create<br/> 6: 2w=896 2ov=112 = 1008 // dmu_tx_hold_zap(tx, sa->sa_layout_attr_obj<br/> 8: 2w=896 2ov=112 = 1008 // OI for new llog object<br/> 136: 2w=384 2ov=112 = 496 // quota usr zap<br/> 137: 2w=384 2ov=112 = 496 // quota grp zap<br/> 203: 2w=0 2ov=240 = 240 // init new llog<br/> 203: 2w=256 2ov=112 = 368 // write to new llog</p> <p>with dnode accounting moved into ZFS and zap blocksize=16K we can shrink it to 2016K,<br/> but it's still a lot. especially when we need to modify few dozen llogs at unlink. say, 128 stripes<br/> would need 128*2=256MB reserved on disk and in memory.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19101/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19101/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ead6df2feee9c143b617cb60e50e403c955bd401</p> <p>Landed for 2.9</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22293" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22293</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cbcd75dc816af495b232577b82bbd37da1c829ca</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22293/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22293/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d16c76ea58f70cdac6c0de0e4fdbe5e329951c33</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22296" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22296</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f4123879778e5c94915851240cb2e6d43e1cca2</p> <p>Patch was reverted due to build issues.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/22296/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22296/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f2d4419227806850cee36c54c0028dc43ee02b1f</p> <p>Landed for 2.9</p> <p>Jenkins build failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13:39:05 /tmp/rpmbuild-lustre-jenkins-hHtDucWf/BUILD/lustre-2.7.19.8/lustre/osd-zfs/osd_oi.c: In function 'osd_oi_create': 13:39:05 /tmp/rpmbuild-lustre-jenkins-hHtDucWf/BUILD/lustre-2.7.19.8/lustre/osd-zfs/osd_oi.c:179: error: 'DN_MAX_BONUSLEN' undeclared (first use in this function) 13:39:05 /tmp/rpmbuild-lustre-jenkins-hHtDucWf/BUILD/lustre-2.7.19.8/lustre/osd-zfs/osd_oi.c:179: error: (Each undeclared identifier is reported only once 13:39:05 /tmp/rpmbuild-lustre-jenkins-hHtDucWf/BUILD/lustre-2.7.19.8/lustre/osd-zfs/osd_oi.c:179: error: for each function it appears in.) 13:39:05 make[7]: *** [/tmp/rpmbuild-lustre-jenkins-hHtDucWf/BUILD/lustre-2.7.19.8/lustre/osd-zfs/osd_oi.o] Error 1 </pre> </div></div> <p><a href="https://build.hpdd.intel.com/job/lustre-b_ieel-reviews/4526/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b_ieel-reviews/4526/</a><br/> <a href="https://build.hpdd.intel.com/job/lustre-b_ieel-reviews/arch=x86_64,build_type=server,distro=el6.8,ib_stack=inkernel/4520/consoleFull" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b_ieel-reviews/arch=x86_64,build_type=server,distro=el6.8,ib_stack=inkernel/4520/consoleFull</a></p> Blocker LU-7895 Related Development Rank 1|hzy58v: Rank (Obsolete) 9223372036854775807