https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8916 Lustre <p>I mounted a ZFS backed lustre target thru the ZPL and it appears that ZAP sizes are not being adjusted as child objects are added or removed. Examples below.</p> <p>Looking at update_log_dir, performing a ls -l yields the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>drw-r--r--. 2 root root    2 Dec 31  1969 update_log_dir </pre> </div></div> <p>But, that's wrong. We should be seeing a size of 3 as an ls shows us there are 3 children.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-bash-4.1# ls -a update_log_dir . .. [0x200000400:0x1:0x0] </pre> </div></div> <p>Looking at oi.10, ls -l displays a size of 0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>drwxr-xr-x. 0 root root 0 Dec 31 1969 oi.10 </pre> </div></div> <p>But, the size we should be seeing is 12 as per the ls below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-bash-4.1# ls -a oi.10 . 0x20000000a:0x0:0x0 0xa:0x3:0x0 0xa:0x5:0x0 0xa:0x7:0x0 0xa:0x9:0x0 .. 0xa:0x0:0x0 0xa:0x4:0x0 0xa:0x6:0x0 0xa:0x8:0x0 0xa:0xa:0x0 </pre> </div></div> <p>In the ZPL, the size of a directory should be the number of objects contained within it. The osd-zfs is clearly breaking that.</p> <p>As a practical use case, in <a href="https://jira.whamcloud.com/browse/LU-8753" title="Recovery already passed deadline with DNE" class="issue-link" data-issue-key="LU-8753"><del>LU-8753</del></a>, we couldn't remove update_log_dir since the directory didn't qualify as empty with a size of 1. The ZPL expects an empty directory to have a size of 2 (for '.' and '..').</p> LU-8916

ods-zfs doesn't manage ZAP sizes correctly

Bug Major Resolved Not a Bug Alex Zhuravlev Giuseppe Di Natale llnl patch Wed, 7 Dec 2016 18:03:55 +0000 Tue, 23 Jan 2018 17:56:12 +0000 Mon, 22 Jan 2018 22:22:03 +0000 0 7 <p>I'll be submitting a preliminary patch shortly. I also have a few questions I need help with.</p> <p>Giuseppe Di Natale (dinatale2@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/24207" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24207</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8916" title="ods-zfs doesn&#39;t manage ZAP sizes correctly" class="issue-link" data-issue-key="LU-8916"><del>LU-8916</del></a> osd-zfs: Correct ZAP size accounting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f6def52b783a18225183e6aa5fdf7d3f5112295</p> <p>Note that the patch does not fix all cases. Joe will submit questions as well.</p> <p>Nathaniel</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Note that the OIs are ZAPs, but are not strictly real directories because entries there do not increase the nlink count on the files/dirs that they are referencing. I definitely agree that being able to view and possibly remove entries from the OI but if that also decrements the dnode link count then this could cause potential issues if unlinking FID files from the OI ZAP via ZPL mountpoint.</p> <p>I've not been looking at the link count too closely yet. I was going to look at that bit later. I'm strictly talking about the size of a directory as reported by ZPL. It is quite possible that I'm missing part of the picture and I should start paying attention to the link count.</p> <p>The OI files explicitly do NOT track the link count themselves (for subdirectories) or on the referenced inodes (for regular files) in order to not upset the visible nlink counter on the inode under normal usage.</p> <p>I measured the patch against zfs performance branch:<br/> before:<br/> mdt 1 file 3000000 dir 16 thr 16 create 306471.11 [ 240979.28, 357970.29] <br/> after:<br/> mdt 1 file 3000000 dir 16 thr 16 create 267600.94 [ 999.97, 347099.55] </p> <p>going to present the result to ZFS guys ...</p> <p>this is what Matt suggested:<br/> 1. Make Lustre not update SA_ZPL_SIZE.<br/> 2. Make the ZPL accept an incorrect SA_ZPL_SIZE without panicking. The ZPL will set z_size based on zap_count(), rather than on SA_ZPL_SIZE.<br/> 3. To maintain backwards compatibility, make the ZPL continue to write an accurate SA_ZPL_SIZE (based on the z_size which will still be accurate due to (2).</p> <p>I've been working on the patch to ZFS..</p> <p>also, can you please clarify the last statement in the description:<br/> As a practical use case, in <a href="https://jira.whamcloud.com/browse/LU-8753" title="Recovery already passed deadline with DNE" class="issue-link" data-issue-key="LU-8753"><del>LU-8753</del></a>, we couldn't remove update_log_dir since the directory didn't qualify as empty with a size of 1. The ZPL expects an empty directory to have a size of 2 (for '.' and '..').</p> <p>but we do set size to 2 for all regular directories (though don't update it):<br/> Object lvl iblk dblk dsize lsize %full type<br/> 3121 2 128K 16K 11.0K 32K 100.00 ZFS directory<br/> 192 bonus System attributes<br/> path /ROOT/TOOR<br/> uid 0<br/> gid 0<br/> mode 40755<br/> size 2</p> <p>so you should be possible to remove with ZPL? but check for emptiness won't work correctly and I'm going to fix this in ZPL as Matt suggested.</p> <p>AFAIK Alex is the one working on thos</p> <p>a short update.. so with the uptodate ZFS (0.7) it's possible to remove such a directory. actually it's possible to remove non-empty directory as well (like CONFIGS/), which I guess is not very nice. as mentioned before, the suggested solution is to let ZFS check a counter in ZAP structure. <br/> two options have been considered:<br/> a) check ZAP's internal counter in zfs_dirempty() &#8211; Matt was fine with that, but that would be extra cost to access that structure (dnode# -&gt; dnode_t structure translation) or we'd need to keep a reference to dnode_t in znode structure<br/> b) wait until SA becomes less expensive, I believe there is a common wish to going this way</p> <p>Olaf to correspond with Alex to understand options better and decide what further work is necessary.</p> <p>Alex,</p> <blockquote> <p>a) check ZAP's internal counter in zfs_dirempty() – Matt was fine with that, but that would be extra cost to access that structure (dnode# -&gt; dnode_t structure translation) or we'd need to keep a reference to dnode_t in znode structure</p></blockquote> <p>If the ZAP's internal counter is only consulted when <em>dzp-&gt;z_size != 2</em> then the cost is only paid when the rmdir would normally fail anyway, right? Or are you thinking of always checking both <em>dzp-&gt;z_size</em> and checking the ZAP's internal counter?</p> <p>Can you expand upon option (b) which I take it you prefer? It sounds like you have reason to think SAs will become cheaper to update and access. Is there in-progress work that will make that so? Thanks.</p> <p>sorry for the late reply.<br/> we can't consult with dzp-&gt;z_size if we want to keep performance as cost of z_size maintainance is very high (due to SA expenses).<br/> unfortunately to access ZAP's internal counter we have to lookup dnode_p* structure by dnode number (stored in znode), which isn't free. the good thing is that rmdir isn't that frequent operation. Ideally we should store dnode_p* in znode structure, I'll talk to Matt about this possibility. in the mean time I'm going to have a patch adding that check for the internal counter. this should fix your issue with silent removal of non-empty directories, correct me If I'm wrong please.</p> <p>I believe that <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a> has pushed a proposed change to ZoL - <a href="https://github.com/zfsonlinux/zfs/pull/6987" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/6987</a></p> <p> </p> <p>ZoL PR 6987 was replaced with<br/> <a href="https://github.com/zfsonlinux/zfs/pull/7019" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/7019</a></p> <p>which was landed to ZoL master and queued for inclusion in zfs-0.7.6 when it is tagged.</p> <p>Fixed in ZFS</p> Related LU-8753 Development Rank 1|hzyxrb: Rank (Obsolete) 9223372036854775807 Severity