https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2619 Lustre <p>When running <tt>lfs quota -u &lt;USER&gt; &lt;FS&gt;</tt> on Sequoia, a couple users do not have any files in their directory but quota reports a bogus large value:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs quota -u pjmccart /p/ls1 Disk quotas for user pjmccart (uid 8624): Filesystem kbytes quota limit grace files quota limit grace /p/ls1 913 0 0 - 18446744073709547865 0 0 - # du -sh /p/ls1/pjmccart/ 913K /p/ls1/pjmccart/ # ls -alR /p/ls1/pjmccart/ /p/ls1/pjmccart/: total 1214 913 drwx------ 2 pjmccart pjmccart 934400 Nov 15 10:28 ./ 302 drwxr-xr-x 2193 root root 308736 Jan 11 08:05 ../ </pre> </div></div> <p>Using <tt>systemtap</tt> to print the <tt>obd_quotactl</tt> structure when the <tt>osc_quotactl</tt> function returns, I see odd values coming from two of the OSCs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc_quotactl: "ls1-OST0037-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=0, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} osc_quotactl: "ls1-OST0038-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551615, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} osc_quotactl: "ls1-OST0039-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=0, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc_quotactl: "ls1-OST0073-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=3, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} osc_quotactl: "ls1-OST0074-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551615, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} osc_quotactl: "ls1-OST0075-osc-c0000003c865a400": {.qc_cmd=8388867, .qc_type=0, .qc_id=8624, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=3, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .dqb_padding=0}} </pre> </div></div> <p>Specifically, the values of <tt>dqb_curinodes</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ls1-OST0074-osc-c0000003c865a400:dqb_curinodes=18446744073709551615 ls1-OST0038-osc-c0000003c865a400:dqb_curinodes=18446744073709551615 </pre> </div></div> LU-2619

Bogus value of dqb_curinodes returned by osc_quotactl

Bug Minor Resolved Won't Fix Niu Yawei Prakash Surya llnl Tue, 15 Jan 2013 11:03:05 +0000 Mon, 11 May 2015 21:10:14 +0000 Tue, 20 Jan 2015 23:38:31 +0000 Lustre 2.4.0 0 5 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I suppose the backend is zfs, and the "quota_iused_estimate" is 0 (you can check it under the osd-zfs proc dir), right?</p> <p>The curinodes from OSTs will not be counted in the total inode usage at the end, though the number implies something is wrong. I guess the MDC was getting the same number too, is it possible to get some log with D_QUOTA enabled on the MDT or the OSTs which returning invalid number? Thanks.</p> <p>Ping. We're still suffering from this.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sierra38 /root &gt; cat /proc/fs/lustre/version lustre: 2.1.4 kernel: patchless_client build: 2.1.4-5chaos-5chaos--PRISTINE-2.6.32-358.11.1.2chaos.ch5.1.x86_64 # sierra38 /root &gt; cat TOSS-27.stp probe module("lquota").function("client_quota_ctl").return { printf("%s: %s: %s\n", probefunc(), @cast($exp, "obd_export")-&gt;exp_obd-&gt;obd_name$, $oqctl$$); } # sierra38 /root &gt; stap TOSS-27.stp -c "lfs quota -u richmond /p/lscratche" Disk quotas for user richmond (uid 1098): Filesystem kbytes quota limit grace files quota limit grace /p/lscratche 8 0 0 - 18446744073709551606 0 0 - client_quota_ctl: "lse-MDT0000-mdc-ffff8802fa988c00": {.qc_cmd=8388615, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=0, .dqb_btime=0, .dqb_itime=0, .dqb_valid=0, .padding=0}} client_quota_ctl: "lse-OST0001-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=0, .dqb_btime=0, .dqb_itime=0, .dqb_valid=0, .padding=0}} client_quota_ctl: "lse-OST0002-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551419, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0003-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551442, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0004-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551404, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0005-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551487, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0006-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551433, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0007-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551521, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0008-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551422, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0009-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551416, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000a-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551325, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000b-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551499, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000c-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551469, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000d-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551340, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000e-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551387, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST000f-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551435, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-OST0010-osc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=18446744073709551326, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} client_quota_ctl: "lse-MDT0000-mdc-ffff8802fa988c00": {.qc_cmd=8388867, .qc_type=0, .qc_id=1098, .qc_stat=0, .qc_dqinfo={.dqi_bgrace=0, .dqi_igrace=0, .dqi_flags=0, .dqi_valid=0}, .qc_dqblk={.dqb_bhardlimit=0, .dqb_bsoftlimit=0, .dqb_curspace=0, .dqb_ihardlimit=0, .dqb_isoftlimit=0, .dqb_curinodes=0, .dqb_btime=0, .dqb_itime=0, .dqb_valid=15, .padding=0}} </pre> </div></div> <blockquote> <p>I suppose the backend is zfs, and the "quota_iused_estimate" is 0 (you can check it under the osd-zfs proc dir), right?</p></blockquote> <p>Yes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># porteri /root &gt; pdsh -w porter[1-16] 'cat /proc/fs/lustre/osd-zfs/*/quota_iused_estimate' | dshbak -c ---------------- porter[1-16] ---------------- 0 </pre> </div></div> <blockquote> <p>The curinodes from OSTs will not be counted in the total inode usage at the end, though the number implies something is wrong. I guess the MDC was getting the same number too, is it possible to get some log with D_QUOTA enabled on the MDT or the OSTs which returning invalid number? Thanks.</p></blockquote> <p>I'll see if I can get some debug logs from the problem OSTs.</p> <p>What's exact client &amp; server version? client_quota_ctl() should only exist in 2.1 and earlier version.</p> <p>Sorry I haven't gotten any debug data pushed to you yet. Would that still be useful? And if so, from the MDT, OST, or both?</p> <p>The client version taken from the previous comment:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sierra38 /root &gt; cat /proc/fs/lustre/version lustre: 2.1.4 kernel: patchless_client build: 2.1.4-5chaos-5chaos--PRISTINE-2.6.32-358.11.1.2chaos.ch5.1.x86_64 </pre> </div></div> <p>The server version:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># porter-mds1 /root &gt; cat /proc/fs/lustre/version lustre: 2.4.0 kernel: patchless_client build: 2.4.0-15chaos-15chaos--PRISTINE-2.6.32-358.14.1.2chaos.ch5.1.1.x86_64 </pre> </div></div> <p>The bugous value returned from OST, the log on OST would be helpul, however, these inode usage values from OSTs will not be accounted, so I guess the value displayed in 'lfs quota' output comes from MDT. Could you get logs from both MDT &amp; OST? Thanks.</p> <p>I have attached the MDS logs generated after running this command:</p> <ol> <li>oslic1 /root &gt; lfs quota -u 40186 /p/lscratche<br/> Disk quotas for user 40186 (uid 40186):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /p/lscratche 14 0 0 - 18446744073709551606 0 0 -</li> <li>oslic1 /root &gt;</li> </ol> <p>The OSS logs total 100MB compressed, so I can't upload them all at once. If you want specific ones, or have an FTP site, I can get them to you.</p> <p>Marc</p> <p>I have privately emailed you details of how to get us the logs</p> <p>Peter</p> <p>Thank you Peter. I created an <a href="https://jira.whamcloud.com/browse/LU-2619" title="Bogus value of dqb_curinodes returned by osc_quotactl" class="issue-link" data-issue-key="LU-2619"><del>LU-2619</del></a> directory, and placed a file called "porter_lustre_logs.tgz" It is about 115MB compressed. Untarred, it will be about 2.4GB. Let me know if you need anything else.</p> <p>Unfortunately, I didn't find any clue in the logs.</p> <p>Marc, could you apply the debug patch (<a href="http://review.whamcloud.com/#/c/8191/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8191/</a>) and try to get the log in following steps again? Thanks a lot.</p> <p>1. 'lctl clear' on MDS and OSTs to clear the debug buffer;<br/> 2. enable D_QUOTA &amp; D_TRACE on MDS and OSTs; (echo +quota &gt; /proc/sys/lnet/debug; echo +trace &gt; /proc/sys/lnet/debug);<br/> 3. 'lctl debug_daemon start $tmpfile 300' on MDS and OSTs to start the debug daemon;<br/> 4. 'lctl mark "======= lfs quota ======"' on MDS and OSTs to set a marker in debug log;<br/> 5. execute the lfs quota command which prints the bugous value;<br/> 6. 'lctl debug_daemon stop' on MDS and OSTs to stop debug daemon;<br/> 7. 'lctl debug_file $tmpfile $logfile' to convert binary logs into text files;<br/> 8. put the text log files in previoius step on ftp;</p> <p>Applying a patch will take a few weeks. Is it useful to run the above steps without the patch?</p> <p>It's better to apply the patch or you can use system tap to print these values in odd_acct_index_lookup(). </p> <p>Sigh.. Systemtap is failing me..</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># porter34 /root &gt; stap -v TOSS-27.stp Pass 1: parsed user script and 91 library script(s) using 101264virt/26156res/2964shr/24060data kb, in 110usr/0sys/111real ms. semantic error: failed to retrieve location attribute for 'osd' (dieoffset: 0x8434c): identifier '$osd' at TOSS-27.stp:3:9 source: @cast($osd, "osd_device")-&gt;od_svname$, ^ Pass 2: analyzed script: 1 probe(s), 7 function(s), 0 embed(s), 0 global(s) using 249680virt/34380res/6848shr/27132data kb, in 20usr/20sys/40real ms. Pass 2: analysis failed. Try again with another '--vp 01' option. # porter34 /root &gt; cat TOSS-27.stp probe module("osd-zfs").function("osd_acct_index_lookup").return { printf("%s: %s: id: %s, ispace: %u, bspace: %u\n", probefunc(), @cast($osd, "osd_device")-&gt;od_svname$, $buf, @cast($rec, "lquota_acct_rec")-&gt;ispace, @cast($rec, "lquota_acct_rec")-&gt;bspace); } </pre> </div></div> <p>This problem happened on both 2.1.4 client 2.4 client, right? Is it possible to make a reproducer?</p> <p>Yes, we see the same behavior on 2.1 and 2.4 clients. The server is 2.4 only, though. I don't know if the same would happen on a 2.1 server. We have a reproducer, but I think it is dependent on the server returning a "bad" value. Perhaps we can try to reproduce this in a VM setup, using a "fail_loc" on the server to return a bogus value to the client? I haven't tried that, but it <em>might</em> work.</p> <p>what do you mean using a 'fail_loc' on server to return a bogus value to client? I don't think server is expected to return a bad value.</p> <p>Well, I'm still unsure where the bad value is coming from, but my guess is it's coming from the server. I could be wrong, though.</p> <p>I'm assuming the bad "dqb_curinodes" values uncovered by the client systemtap script is coming from the server, is that not the case?</p> <blockquote> <p>I'm assuming the bad "dqb_curinodes" values uncovered by the client systemtap script is coming from the server, is that not the case?</p></blockquote> <p>That's quite possible. I'm just not sure the purpose of using a 'fail_loc' on server to return bad value to client.</p> <p>The bogus 'dqb_curinodes' comes from OST, I'm wondering how it can contribute to the 'files' of 'lfs quota' output, because we only collect inode usage on MDTs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* collect space usage from OSTs */</span> oqctl_tmp-&gt;qc_dqblk.dqb_curspace = 0; rc = obd_quotactl(sbi-&gt;ll_dt_exp, oqctl_tmp); <span class="code-keyword">if</span> (!rc || rc == -EREMOTEIO) { oqctl-&gt;qc_dqblk.dqb_curspace = oqctl_tmp-&gt;qc_dqblk.dqb_curspace; oqctl-&gt;qc_dqblk.dqb_valid |= QIF_SPACE; } <span class="code-comment">/* collect space &amp; inode usage from MDTs */</span> oqctl_tmp-&gt;qc_dqblk.dqb_curspace = 0; oqctl_tmp-&gt;qc_dqblk.dqb_curinodes = 0; rc = obd_quotactl(sbi-&gt;ll_md_exp, oqctl_tmp); <span class="code-keyword">if</span> (!rc || rc == -EREMOTEIO) { oqctl-&gt;qc_dqblk.dqb_curspace += oqctl_tmp-&gt;qc_dqblk.dqb_curspace; oqctl-&gt;qc_dqblk.dqb_curinodes = oqctl_tmp-&gt;qc_dqblk.dqb_curinodes; oqctl-&gt;qc_dqblk.dqb_valid |= QIF_INODES; } <span class="code-keyword">else</span> { oqctl-&gt;qc_dqblk.dqb_valid &amp;= ~QIF_SPACE; } </pre> </div></div> <p>I did some local testing that making OST to return fake 'curinodes' to client, however, client ignored the fake value as expected.</p> <p>While investigating why server returns bogus value, I think I'd verify the client code you running wasn't changed by some unexpected patch. Could you show me where to check the client code? (llnl tree? which tag?) Thank you.</p> <p>As always, our source and releases are on github: <a href="https://github.com/chaos/lustre" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre</a></p> <p>As far as which releases were installed on the servers and clients in question, I'll have to ask the admins. Marc Stearman, can you double check this issue is still occurring on Sequoia and report back the version currently installed there?</p> <p>Yes it is still happening on all of our file systems, so the current tag will work for you.</p> <p>In other words, tag 2.4.2-13chaos.</p> <p>Thank you, I didn't see any difference in client code.</p> <p>Looks the debug patch (<a href="http://review.whamcloud.com/#/c/8191/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8191/</a>) has been applied in the code, is it possible to capture server logs with D_QUOTA enabled? So we can see if the bogus value is returned from osd_acct_index_lookup().</p> <p>I enabled +quota debugging on the MDS. Then I ran this command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@surface86:~]# lfs quota -u weems2 /p/lscratche Disk quotas for user weems2 (uid 59519): Filesystem kbytes quota limit grace files quota limit grace /p/lscratche 88 0 0 - 18446744073709551462 0 0 - [root@surface86:~]# </pre> </div></div> <p>You can see that the files column is very large. I then dumped the debug logs on the MDS right after that: These are the lines from the quota debugging:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00040000:04000000:5.0F:1421104120.034812:0:13022:0:(qmt_handler.c:65:qmt_get()) $$$ fetch settings qmt:lse-QMT0000 pool:0-md id:59519 enforced:0 hard:0 soft:0 granted:0 time:0 qunit:0 edquot:0 may_rel:0 revoke:0 00040000:04000000:5.0:1421104120.034818:0:13022:0:(qmt_handler.c:65:qmt_get()) $$$ fetch settings qmt:lse-QMT0000 pool:0-dt id:59519 enforced:0 hard:0 soft:0 granted:0 time:0 qunit:0 edquot:0 may_rel:0 revoke:0 00000001:04000000:4.0F:1421104120.068222:0:13022:0:(osd_quota.c:122:osd_acct_index_lookup()) lse-MDT0000: id:e87f, ispace:18446744073709551462, bspace:90112 </pre> </div></div> <p>Do you need lines from OSTs as well, or just from the MDS?</p> <p>From the log we can see the bogus value is from MDS, and it's read from the zap object which we created for inode accounting. Given that this problem happened only for inode accounting, I highly suspect it's related to <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a>. I think a temporary workaround is to set the quota_iused_estimate to 1.</p> <blockquote> <p>Do you need lines from OSTs as well, or just from the MDS?</p></blockquote> <p>No, I think MDS log is enough. Thank you.</p> <p>Thanks, setting quota_iused_estimate to 1 reports a more realistic value:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@surface86:~]# lfs quota -u weems2 /p/lscratche Disk quotas for user weems2 (uid 59519): Filesystem kbytes quota limit grace files quota limit grace /p/lscratche 88 0 0 - 1 0 0 - [root@surface86:~]# </pre> </div></div> <p>I will watch the status of <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a>. Are there any other downsides to setting quota_iused_estimate to 1?</p> <blockquote> <p>Are there any other downsides to setting quota_iused_estimate to 1</p></blockquote> <p>With quota_iused_estimate = 1, the reported inode accounting is calculated based on consumed space on MDT, it's not accurate as quota_iused_estimate = 0.</p> <p>I think this is a decent workaround for my purposes. I'm happy to close this one, and I will await a fix in <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a> that will allow us to scan the file system and fix broken ZAP entries.</p> <p>ok thanks Marc!</p> Related LU-2435 Development Rank 1|hzvfav: Rank (Obsolete) 6124 Severity