https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4110 Lustre <p>Hello,</p> <p>we have a problem with our new lustre installation.<br/> We found that, for some unknown reason the quota on a single OST (OST001b) seems corrupt, we were generating files in a loop for testing, and the quota was hit much sooner than anticipated or set, but only on some files.<br/> For some unknown reason, the quota on a single OST (OST001b) is corrupt, we have also tried to completely turn off quotas for user and group and turn it on again, but It did not help as well.</p> <p>How can we recover OST-0001b`s quota?</p> e2fsprogs-1.42.7.wc1-7.el6.x86_64<br/> e2fsprogs-devel-1.42.7.wc1-7.el6.x86_64<br/> e2fsprogs-static-1.42.7.wc1-7.el6.x86_64<br/> e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64<br/> LU-4110

Odd quota on single OST

Bug Critical Resolved Incomplete Niu Yawei Matteo Piccinini Wed, 16 Oct 2013 08:51:55 +0000 Fri, 14 Mar 2014 01:44:02 +0000 Fri, 14 Mar 2014 01:44:02 +0000 Lustre 2.4.0 0 5 <p><span class="error">&#91;root@node123&#93;</span># lfs quota -u eric -v /cluster/scratch_xp<br/> Disk quotas for user eric (uid 804):<br/>     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace<br/> /cluster/scratch_xp 145400  2147483648 2147483648       -   36136  209715  209715       -<br/> prism-MDT0000_UUID 1712       -       0       -   36136       -   65536       -<br/> prism-OST0000_UUID  4516       - 16777216       -       -       -       -       -<br/> prism-OST0001_UUID  4516       - 16777216       -       -       -       -       -<br/> prism-OST0002_UUID  4520       - 16777216       -       -       -       -       -<br/> prism-OST0003_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST0004_UUID  4508       - 16777216       -       -       -       -       -<br/> prism-OST0005_UUID 4512       - 16777216       -       -       -       -       -<br/> prism-OST0006_UUID  4520       - 16777216       -       -       -       -       -<br/> prism-OST0007_UUID 4512       - 16777216       -       -       -       -       -<br/> prism-OST0008_UUID 4512       - 16777216       -       -       -       -       -<br/> prism-OST0009_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST000a_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST000b_UUID  4512       - 16777216       -       -       -       -       -<br/> prism-OST000c_UUID  4516       - 16777216       -       -       -       -       -<br/> prism-OST000d_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST000e_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST000f_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0010_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0011_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST0012_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0013_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0014_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0015_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0016_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST0017_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST0018_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST0019_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST001a_UUID 4512       - 16777216       -       -       -       -       -<br/> prism-OST001b_UUID 3680*      -    3680       -       -       -       -       -<br/> prism-OST001c_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST001d_UUID 4516       - 16777216       -       -       -       -       -<br/> prism-OST001e_UUID 4520       - 16777216       -       -       -       -       -<br/> prism-OST001f_UUID 4520       - 16777216       -       -       -       -       -</p> <p>root@node123]# perl -e 'printf("p-oss0%d\n",0x1b%8+1)'<br/> p-oss04</p> <p>I did not observe anything unusual in the /var/log/messages files of p-oss04.</p> <p><span class="error">&#91;root@p-oss04 quota_slave&#93;</span># grep -wA1 804 /proc/fs/lustre/osd-ldiskfs/prism-OST001b/quota_slave/limit_user<br/> limit_user:- id: 804<br/> limit_user- limits: </p> { hard: 2147483648, soft: 2147483648, granted: 0, time: 0 } <p><span class="error">&#91;root@p-oss04 quota_slave&#93;</span># cat acct_user<br/> usr_accounting:</p> <ul class="alternate" type="square"> <li>id: 0<br/> usage: { inodes: 3557, kbytes: 1028 }</li> <li>id: 804<br/> usage: { inodes: 1132, kbytes: 3680 }</li> </ul> <p>I have also tried to completely turn off quotas for user and group and turn it on again. It did<br/> not help as well:</p> <p><span class="error">&#91;root@p-mds1 ~&#93;</span># lctl get_param osd-<b>.</b>.quota_slave.info<br/> osd-ldiskfs.prism-MDT0000.quota_slave.info=<br/> target name: prism-MDT0000<br/> pool ID: 0<br/> type: md<br/> quota enabled: ug<br/> conn to master: setup<br/> space acct: ug<br/> user uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span><br/> group uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span><br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl conf_param prism.quota.ost=none<br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl conf_param prism.quota.mdt=none<br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl get_param osd-<b>.</b>.quota_slave.info<br/> osd-ldiskfs.prism-MDT0000.quota_slave.info=<br/> target name: prism-MDT0000<br/> pool ID: 0<br/> type: md<br/> quota enabled: none<br/> conn to master: setup<br/> space acct: ug<br/> user uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span><br/> group uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span><br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl conf_param prism.quota.mdt=ug<br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl conf_param prism.quota.ost=ug<br/> <span class="error">&#91;root@p-mds1 ~&#93;</span># lctl get_param osd-<b>.</b>.quota_slave.info<br/> osd-ldiskfs.prism-MDT0000.quota_slave.info=<br/> target name: prism-MDT0000<br/> pool ID: 0<br/> type: md<br/> quota enabled: ug<br/> conn to master: setup<br/> space acct: ug<br/> user uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span><br/> group uptodate: glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span></p> <p>How can I recover OST-0001b`s quota?</p> <p>Could you collect log (with D_QUOTA enabled) for OST-0001b and MDT? Thanks.</p> <ul class="alternate" type="square"> <li>Enable D_QUOTA by "echo +quota &gt; /proc/sys/lnet/debug" on OSS (which OST-0001b located) and MDS;</li> <li>lctl debug_daemon start tmpfile 500; (on OSS &amp; MDS)</li> <li>lctl mark "=== start test"; (on OSS &amp; MDS)</li> <li>Write to the file which is striped on OST-0001b as user eric; (on client)</li> <li>lctl debug_daemon stop; (on OSS &amp; MDS)</li> <li>lctl debug_file tmpfile logfile; (on OSS &amp; MDS)</li> </ul> <p>Hope we can find something useful from the logfiles.</p> <p>logs: <a href="ftp://ftp.whamcloud.com/uploads/LU-4110" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/LU-4110</a></p> <p>I didn't find anything abnormal in the logs, and the test passed successfully. Maybe there was some connection problem between OST-0001b with MDT?</p> <p>Please keep reporting if you find anything wrong. Thank you.</p> <p>Hello Matteo, <br/> We have not heard from you in a while on this issue. <br/> Can I mark it as resolved? <br/> Thanks,<br/> ~ jfc.</p> <p>Seems like this may have been a transient problem.<br/> We can reopen this again if the problem recurs.<br/> ~ jfc.</p> Development Rank 1|hzw5tj: Rank (Obsolete) 11066 Severity