https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5309 Lustre <p>It looks as though the <tt>zfs userspace</tt> command is completely broken on at least one of our ZFS datasets backing a Lustre OST running a Lustre 2.4.2 based release.</p> <p>Here's what <tt>zfs list</tt> shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pilsner1 /root &gt; zfs list -t all NAME USED AVAIL REFER MOUNTPOINT pilsner1 17.3T 46.9T 30K /pilsner1 pilsner1/lsd-ost0 17.3T 46.9T 15.9T /pilsner1/lsd-ost0 pilsner1/lsd-ost0@17jun2014 1.39T - 10.6T - </pre> </div></div> <p>And here's what <tt>zfs userspace</tt> shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pilsner1 /root &gt; zfs userspace pilsner1/lsd-ost0@17jun2014 TYPE NAME USED QUOTA POSIX User &lt;omitted&gt; 21.5K none POSIX User &lt;omitted&gt; 118M none POSIX User &lt;omitted&gt; 6.49G none POSIX User &lt;omitted&gt; 72.1M none POSIX User &lt;omitted&gt; 725M none </pre> </div></div> <p>This is clearly broken, and according to Matt Ahrens:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;+mahrens&gt; prakash: yeah, that pbpaste output looks wrong. All the "REFER" space should be accounted for by "zfs userspace" </pre> </div></div> <p>Additionally, running that command on the dataset itself (as opposed to a snapshot) fails with an error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pilsner1 /root &gt; zfs userspace pilsner1/lsd-ost0 cannot get used/quota for pilsner1/lsd-ost0: dataset is busy </pre> </div></div> <p>This is not the case for filesystems mounted through the ZPL.</p> LU-5309

The `zfs userspace` command is broken on Lustre-ZFS-OST datasets

Bug Minor Resolved Cannot Reproduce Nathaniel Clark Prakash Surya llnl Wed, 9 Jul 2014 16:08:08 +0000 Thu, 12 Feb 2015 18:03:16 +0000 Thu, 12 Feb 2015 18:03:16 +0000 Lustre 2.4.2 0 5 <p>Nathaniel &#8211; please review.<br/> Thank you,<br/> ~ jfc.</p> <p>Referenced (REFER) space does not equal space used (USED) as reported by "zfs userspace". The latter reports the total space of all the blocks allocated where REFER is (according to the zfs man page) " The amount of data that is accessible by this dataset, which may or may not be shared with other datasets in the pool". Even in a small scale test on a fresh plain ZFS filesystem these numbers don't agree:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ zpool create test /dev/volgrp/logvol $ zfs create test/foo $ mkdir /test/foo/user $ chown user.user /mnt/foo/user $ su user $ echo blah &gt; /mnt/foo/user/bar $ exit $ zfs list -pt all zfs list -pt all NAME USED AVAIL REFER MOUNTPOINT test 188416 1031610368 31744 /test test/foo 50688 1031610368 32256 /test/foo $ zfs userspace -p test/foo TYPE NAME USED QUOTA POSIX User user 2560 none POSIX User root 1536 none </pre> </div></div> <p>So if they are supposed to agree, that isn't a Lustre issue.</p> <p>Well first off, the Lustre zfs-osd and ZFS on Linux ZPL are two completely separate interfaces to the underlying internals of ZFS. So even if the accounting was broken through the ZPL, that does not necessarily mean it's for the same reason as through the zfs-osd. So you can't simply write this off because you think the accounting is incorrect through the posix layer.</p> <p>And second, I don't think they're supposed to be exactly the same. I'd have to study the code to better understand, though, since I'm not very familiar with how the accounting fits together.</p> <p>Being off by a number of megabytes doesn't seem strange to me, being off by number of terabytes does seem strange. Through the ZPL, I see this on a new pool and filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@surya1-guest-1 ~]# zpool create -f tank /dev/vd{b,c,d} [root@surya1-guest-1 ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT tank 93K 23.4G 30K /tank [root@surya1-guest-1 ~]# dd if=/dev/urandom of=/tank/file1 bs=1M count=100 1&gt;/dev/null 2&gt;&amp;1 [root@surya1-guest-1 ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT tank 100M 23.3G 100M /tank [root@surya1-guest-1 ~]# zfs userspace tank TYPE NAME USED QUOTA POSIX User root 100M none [root@surya1-guest-1 ~]# zfs list -p NAME USED AVAIL REFER MOUNTPOINT tank 105055232 25063914496 104970752 /tank [root@surya1-guest-1 ~]# zfs userspace tank -p TYPE NAME USED QUOTA POSIX User root 104942080 none [root@surya1-guest-1 ~]# echo 104970752-104942080 | bc 28672 </pre> </div></div> <p>That's a difference of exactly 28KB. This is in contrast to the example in the description, where the difference is over 10 TB.</p> <p>I believe the <tt>zfs_space_delta_cb</tt> performs this space accounting tracking in the ZPL, and I would not be surprised if we never implemented equivalent functionality for the zfs-osd.</p> <p>Running a similar test on a fresh osd-zfs backed lustre filesystem I also get equivalent accounting as you did (this is with lustre master build 2553 and zfs/spl 0.6.3).</p> <p>I will double check Lustre 2.4.3 and zfs/spl 0.6.1.</p> <blockquote> <p>Running a similar test on a fresh osd-zfs backed lustre filesystem I also get equivalent accounting as you did (this is with lustre master build 2553 and zfs/spl 0.6.3).</p></blockquote> <p>Hmm... that's interesting because I get "more expected" results when running the <tt>zfs userspace</tt> command against one of our osd-zfs filesystems backing our test filesystem.</p> <blockquote> <p>I will double check Lustre 2.4.3 and zfs/spl 0.6.1.</p></blockquote> <p>Yea, perhaps there was issues with the earlier ZFS releases.. The production system I checked (and seemed "broken") was formatted originally with a 2.4.0 based lustre release and zfs 0.6.2, IIRC. While the test filesystem was formatted with a lustre 2.4.2 based release and ZFS 0.6.3.</p> <p>I have not been able to reproduce this on Lustre 2.4.3 with zfs/spl 0.6.1, but it was a freshly formatted system, and I was just moving large files around to see if numbers matched.</p> <p>This is an old and not very critical bug and we have not been able to reproduce it.</p> <p>~ jfc.</p> Development Rank 1|hzwqyv: Rank (Obsolete) 14835 Severity