https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1720 Lustre <p>We set quota "ug3" to all OSTs and MDT, then an also set 5TB quota limitation to a user. But, if user1 writes files to single OST, it exceeds quota limitation when total file size gets 4TB.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs quota -v -u user1 /lustre/ Disk quotas for user user1 (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /lustre/ 4295057504 0 5368709120 - 13 0 0 - lustre-MDT0000_UUID 4 - 1024 - 13 - 0 - lustre-OST0000_UUID 0 - 1024 - - - - - lustre-OST0001_UUID 4295057500* - 4294959104 - - - - - lustre-OST0002_UUID 0 - 1024 - - - - - lustre-OST0003_UUID 0 - 1024 - - - - - .. .. </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param lquota.*.quota_type lquota.lustre-OST0001.quota_type=ug3 lquota.lustre-OST0004.quota_type=ug3 lquota.lustre-OST0008.quota_type=ug3 lquota.lustre-OST000c.quota_type=ug3 lquota.lustre-OST0011.quota_type=ug3 lquota.lustre-OST0015.quota_type=ug3 lquota.lustre-OST0019.quota_type=ug3 lquota.lustre-OST001d.quota_type=ug3 lquota.lustre-OST0021.quota_type=ug3 lquota.lustre-OST0025.quota_type=ug3 lquota.lustre-OST0028.quota_type=ug3 lquota.lustre-OST002d.quota_type=ug3 lquota.lustre-OST0031.quota_type=ug3 lquota.lustre-OST0035.quota_type=ug3 lquota.lustre-OST0039.quota_type=ug3 # lctl get_param lquota.*.quota_type lquota.lustre-MDT0000.quota_type=ug3 </pre> </div></div> CentOS5.8 Lustre-1.8.8-wc1 LU-1720

Quota doesn't work over 4TB on single OST

Bug Major Resolved Fixed Yang Sheng Shuichi Ihara Wed, 8 Aug 2012 02:25:52 +0000 Fri, 24 Nov 2017 16:15:53 +0000 Thu, 1 Nov 2012 02:46:16 +0000 Lustre 1.8.8 Lustre 2.1.4 Lustre 1.8.9 0 9 <p>Hi, Ihara</p> <p>Could you collect the messages on OSTs? I'm afraid that the quota file for local fs (operational quota file) is not coverted to 64bit yet, just like <a href="https://jira.whamcloud.com/browse/LU-1584" title="error set quota fs limit" class="issue-link" data-issue-key="LU-1584"><del>LU-1584</del></a>?</p> <p>Hi Niu, I will send you messages latter (system was shutdown, will bootup soon), but this is completely new test system. The filesystem is formatted with </p> {ost,mdt} <p>.quota_type=ug3.</p> <p>Hi Niu,</p> <p>tested again, it's very simple configuraiton.<br/> 1 x OSS, 1 x OST and 1 x Client. user1's quota limit is 5TB, and write 4.5TB to single OST. Formated with </p> {ost,mdt} <p>.quota_type=ug3, but it hits again when quota is exceeded by 4TB.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs quotacheck -ug /lustre/ # lfs setquota -B 5368709120 -u user1 /lustre # su - user1 user1 writes files 10 x 450GB files to /lustre dd: writing `/lustre/quota_test/file-10': Disk quota exceeded dd: closing output file `/lustre/quota_test/file-10': Input/output error $ lfs quota -v -u user1 /lustre/ Disk quotas for user user1 (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /lustre/ 4294935256 0 5368709120 - 13 0 0 - lustre-MDT0000_UUID 4 - 1024 - 13 - 0 - lustre-OST0000_UUID 4294935252* - 4294933504 - - - - - </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> OSS's messages when quota is exceeded. Aug 8 23:55:51 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Aug 8 23:55:51 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Aug 8 23:55:51 s02 kernel: Lustre: 19756:0:(quota_interface.c:491:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) Aug 8 23:55:52 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Aug 8 23:55:52 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) Skipped 2473 previous similar messages Aug 8 23:55:53 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Aug 8 23:55:53 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) Skipped 1814 previous similar messages Aug 8 23:55:55 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Aug 8 23:55:55 s02 kernel: LustreError: 18960:0:(quota_context.c:685:dqacq_completion()) Skipped 3868 previous similar messages </pre> </div></div> <p>Hi, Ihara</p> <p>Which looks like the same problem reported in the lustre-discuss, and finally, when they format the filesystem with e2fsprogs-1.41.90.wc4, the problem is gone. (they can reproduce the problem with e2fsprogs-1.41.90.wc3). Could you check the e2fsprogs version on server? Thanks.</p> <p>Hi Niu,</p> <p>e2fsprogs-1.42.3.wc1 is installed and the filesystem is formated with it.<br/> I did same testing on the master branch, but the problem didn't happen.</p> <p>It's too bad that the error message only report an error code, could you apply this patch (which print more information) and enable D_QUOTA while running tests? Thanks.</p> <p>print more information when set local limit failed.</p> <p>Niu, could you plesae check patch again? it seems to be failing with patch.</p> <p>/usr/src/lustre-1.8.8/lustre/quota/quota_context.c: In function 'dqacq_completion':<br/> /usr/src/lustre-1.8.8/lustre/quota/quota_context.c:688: warning: format '%d' expects type 'int', but argument 13 has type 'long unsigned int'<br/> make<span class="error">&#91;6&#93;</span>: *** <span class="error">&#91;/usr/src/lustre-1.8.8/lustre/quota/quota_context.o&#93;</span> Error 1<br/> make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;/usr/src/lustre-1.8.8/lustre/quota&#93;</span> Error 2<br/> make<span class="error">&#91;4&#93;</span>: *** <span class="error">&#91;/usr/src/lustre-1.8.8/lustre&#93;</span> Error 2</p> <p>fix the complile error.</p> <p>sorry, please use the updated one.</p> <p>debug log with enabled D_QUOTA </p> <p>syslog on OSS after patches applied.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 9 13:57:37 s02 kernel: LustreError: 8891:0:(quota_context.c:691:dqacq_completion()) error set quota fs limit! rc:-34, count:1024, hardlimit:4294967296 isblk:b Aug 9 13:57:37 s02 kernel: LustreError: 8891:0:(quota_context.c:691:dqacq_completion()) Skipped 115657 previous similar messages Aug 9 13:57:37 s02 kernel: Lustre: 9294:0:(quota_interface.c:475:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) </pre> </div></div> <p>Looks the 'hardlimit' and 'count' values are sane, I suspect that the local quota file was created as 32bit somehow... Could you just mount the ost device as ldiskfs and check the quota file name on it? The name should be "lquota.user" &amp; "lquota.group" for 32bit, or "lquota_v2.user" &amp; "lquota_v2.group" for 64bit. Thanks.</p> <p>Niu,<br/> Yeah, I thought and had checked them before and lquota_v2.</p> {user,group} <p> existed. That was odd and filed this prolbem on here.<br/> Here is I just checked. There are no "lquota.user" &amp; "lquota.group" files, but v2 files are existing, instead.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t ldiskfs /dev/mapper/LUN00 /mnt/lustre/LUN00/ # ls /mnt/lustre/LUN00/ CONFIGS O health_check last_rcvd lost+found lquota_v2.group lquota_v2.user </pre> </div></div> <p>Tested with 1.8.7, but still hit same probolem.</p> <p>I have tested with lustre-2.1.2 on RHEL5, but we hit same quota limitation by 4TB.</p> <p>This is reproducer of this problem.<br/> And here is what I did to make the lustre and tesing.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># MDS # mkfs.lustre --reformat --mgs --mdt --param mdt.quota_type=ug3 /dev/sdb1 # mount -t lustre /dev/sdb1 /mnt/lustre/MDT # lctl get_param lquota.*.quota_type lquota.mdd_obd-lustre-MDT0000.quota_type=ug3 # OSS # mkfs.lustre --reformat --ost --mgsnode=192.168.100.129@o2ib --param ost.quota_type=ug3 /dev/mapper/LUN59 # mount -t lustre /dev/mapper/LUN59 /mnt/lustre/LUN59 # lctl get_param lquota.*.quota_type lquota.lustre-OST0000.quota_type=ug3 # Client # mount -t lustre 192.168.100.129@o2ib:/lustre /lustre # nohup /tmp/reproducer.sh &amp; # lfs quota -u user1 -v /lustre/ Disk quotas for user user1 (uid 1000): Filesystem kbytes quota limit grace files quota limit grace /lustre/ 4295204904 0 5368709120 - 100 0 0 - lustre-MDT0000_UUID 0 - 1024 - 100 - 0 - lustre-OST0000_UUID 4295204904* - 4294966272 </pre> </div></div> <p>Thanks for your update, Ihara!</p> <p>Finally, I found the reason: the kernel quota-large-limits-rhel5.patch (which makes kernel do_set_dqblk() support 64bits) is mis-updated in ba5dd769f66194a80920cf93d6014c78729efaae (<a href="https://jira.whamcloud.com/browse/LU-674" title="Update RHEL5 kernel to 2.6.18-274.3.1.el5 security update" class="issue-link" data-issue-key="LU-674"><del>LU-674</del></a> kernel update RHEL5.7 <span class="error">&#91;2.6.18-274.3.1.el5&#93;</span>).</p> <p>Yangshen, could you take a look on this, and fix the patch? Thanks.</p> <p>Niu, thanks for analysis. what do you mean mis-updated?</p> <p>The patch updated incorrectly when supporting new kernel. Please check: <a href="http://review.whamcloud.com/#change,3599" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3599</a></p> <p>patch for b2_1: <a href="http://review.whamcloud.com/3600" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3600</a></p> <p>patch for b1_8: <a href="http://review.whamcloud.com/3599" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3599</a></p> <p>I reproduced this problem with lustre-1.8.8 on RHEL5 and confirmed it's fixed by kernel patch on <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a>. Also confirmed fixing on lustre-2.1.2 with RHEL5 and the problem doesn't happen on RHEL6 even without patches.</p> <p>Hi, I've confirmed the patch <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a> solves this problem, would you please merge if the review is finished?</p> <p>Landed to b1_8. Still needs to land to 2.x branches</p> <p>Patch landed to all branch. Close bug.</p> Related Development Rank 1|hzv3gv: Rank (Obsolete) 4061 Severity