https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1584 Lustre <p>At an our customer site, an group hit quota limit regardless that group was not set quota limitation yet. (this group's quota size is 160TB, but that group exceeded quota limit around 150TB) In this case, the following messages showed up on an OSS. Is this related to <a href="https://jira.whamcloud.com/browse/LU-935" title="Crash lquota:dquot_create_oqaq+0x28f/0x510" class="issue-link" data-issue-key="LU-935"><del>LU-935</del></a>?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 29 14:58:47 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:47 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 19 previous similar messages Jun 29 14:58:47 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:47 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 542 previous similar messages Jun 29 14:58:48 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:48 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 1774 previous similar messages Jun 29 14:58:49 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:49 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 4177 previous similar messages Jun 29 14:58:51 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:51 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 8905 previous similar messages Jun 29 14:58:55 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:58:55 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 17831 previous similar messages Jun 29 14:59:03 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:59:03 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 42377 previous similar messages Jun 29 14:59:19 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) error set quota fs limit! (rc:-34) Jun 29 14:59:19 nos031i kernel: LustreError: 8322:0:(quota_context.c:685:dqacq_completion()) Skipped 86190 previous similar messages </pre> </div></div> lustre-1.8.7-wc1 RHEL5 LU-1584

error set quota fs limit

Bug Minor Resolved Duplicate Niu Yawei Shuichi Ihara Fri, 29 Jun 2012 07:04:41 +0000 Sat, 18 Aug 2012 10:33:48 +0000 Sat, 18 Aug 2012 10:33:48 +0000 Lustre 1.8.7 0 7 <p>Could you try upgrading to 1.8.8-wc1? It may solve your problem. There have been a number of fixes in quota code including the one from <a href="https://jira.whamcloud.com/browse/LU-935" title="Crash lquota:dquot_create_oqaq+0x28f/0x510" class="issue-link" data-issue-key="LU-935"><del>LU-935</del></a>.</p> <p>Bob,<br/> Yes, we want, but before upgrade we also wanted to make sure this is exactly same problem of <a href="https://jira.whamcloud.com/browse/LU-935" title="Crash lquota:dquot_create_oqaq+0x28f/0x510" class="issue-link" data-issue-key="LU-935"><del>LU-935</del></a>.<br/> Until upgrading, the following workaround would be help to avoid this issue that described on <a href="https://jira.whamcloud.com/browse/LU-935" title="Crash lquota:dquot_create_oqaq+0x28f/0x510" class="issue-link" data-issue-key="LU-935"><del>LU-935</del></a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param lquota.*.quota_switch_qs=0 </pre> </div></div> <p>By the way, where did the 1.8.7 release you are running come from, Whamcloud directly or some other supplier? Can you upgrade to a Whamcloud release or is that path blocked for you?</p> <p>#define ERANGE 34 /* Math result not representable */</p> <p>Looks similar to the problem reported on wc-discuss:<br/> <a href="https://groups.google.com/a/whamcloud.com/group/wc-discuss/browse_thread/thread/91b5ceae1a663bad/01a84d777079cc46?utoken=8NerwC0AAADp76oaNbE87RxLb3AWLsFF1dK1zSSxgp828G_EjM7-wrBMBmDPXU48RC1TYmZ5gaU#01a84d777079cc46" class="external-link" target="_blank" rel="nofollow noopener">https://groups.google.com/a/whamcloud.com/group/wc-discuss/browse_thread/thread/91b5ceae1a663bad/01a84d777079cc46?utoken=8NerwC0AAADp76oaNbE87RxLb3AWLsFF1dK1zSSxgp828G_EjM7-wrBMBmDPXU48RC1TYmZ5gaU#01a84d777079cc46</a></p> <p>Johann,<br/> Originally, we hit a quota prolbem (<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>). We did setquota zero to all groups, but <a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a> still can't be fixed yet. Niu is investigating this and grateful advices us.<br/> Two days ago, we hit "quota exceeded" for an group, but this quota's limit is more high and shouldn't be exceeded size.</p> <p>Yesterday, we did setquota zero to this group, but today hit "quota exceeded" again for same group and showed up this messages on an OSS.</p> <p>Niu</p> <p>Are you able to help with this one?</p> <p>Peter</p> <p>PS/ Added Johann as a watcher so that he can see the feedback from Ihara</p> <p>could you run 'lfs quota -v -u $group_id $fsname' and post the output here? I don't see why OST still acquire quota from master after the limit is cleared.</p> <p>And the setquota on local fs got ERANAGE, it looks like the problem the Johann mentioned, could you verify what version of e2fsprogs is using on servers? Thanks.</p> <p>it seems that quota was disabled after this issue was showed up. let me try to get "lfs quota -v -g ..".<br/> btw, e2fsprogs-1.41.90.wc3 is running on here.</p> <p>The similar problem reported in wc-discuss is also reproduced by e2fsprogs-1.41.90.wc3, but I don't see how e2fsprogs can affect the quota files. BTW, could you also check the 'quota_type' for all servers? (/proc/fs/lustre/obdfilter/*/quota_type). Thanks.</p> <p>Niu, could you please summarize the steps you used to reproduce the problem? Could you please also attach the output of e2fsck? TIA</p> <p>I didn't reporduce it, I just searched the wc-disucss.</p> <p>as far as I saw wc-discuss, finally he reformatted the filesystem with e2fsprogs-1.41.90.wc4 and confirmed the problem is gone. But, for existing filesystem, how can we fix this? just upgrade e2fsprogs to 1.41.90.wc4? the larger quota limit (175921860444160) doesn't matter?</p> <p>I didn't think the problem is caused by e2fsprogs-1.41.90.wc3, and I tried it in my local test environment, it can't be reproduced.</p> <p>The 160TB limit should be an valid limit for the v2 qutoa file. Could you paste the output of "quota_type" &amp; "lfs quota -v -g ..." here? Thanks.</p> <p>Niu,</p> <p>Uploaded on uploads/<a href="https://jira.whamcloud.com/browse/LU-1584" title="error set quota fs limit" class="issue-link" data-issue-key="LU-1584"><del>LU-1584</del></a>.</p> <p>we found a related problem. </p> <p>quota_type was set as "ug1" even if we set "ug2" during the format time.<br/> We are setting "ost.quota_type=ug2" when we formatted OST/MDT.</p> <p>Here is quick reproducer what I said.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkfs.lustre --reformat --mgs --mdt --param mdt.quota_type=ug2 /dev/sda1 # mkfs.lustre --reformat --mgsnode=s01i@o2ib --ost --param ost.quota_type=ug2 /dev/sda2 # mkfs.lustre --reformat --mgsnode=s01i@o2ib --ost --param ost.quota_type=ug3 /dev/sda3 # mkfs.lustre --reformat --mgsnode=s01i@o2ib --ost /dev/sda4 # for i in `seq 1 4`; do mount -t lustre /dev/sda$i /mnt/lustre/sda$i; done </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param lquota.*.quota_type lquota.lustre-MDT0000.quota_type=2 lquota.lustre-OST0000.quota_type=1 lquota.lustre-OST0001.quota_type=3 lquota.lustre-OST0002.quota_type=3 # tunefs.lustre --dryrun /dev/sda2 checking for existing Lustre data: found CONFIGS/mountdata Reading CONFIGS/mountdata Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=192.168.100.150@o2ib ost.quota_type=ug2 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: errors=remount-ro,extents,mballoc Parameters: mgsnode=192.168.100.150@o2ib ost.quota_type=ug2 exiting before disk write. </pre> </div></div> <p>OST0000's (/dev/sda2) quota_type is ug1 even it's formatted with ug2 and parameter is set on OST.</p> <p>Just try removing ost.quota_type=ug2, but it's still ug1.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># umount /dev/sda2 # tunefs.lustre --erase-param /dev/sda2 # tunefs.lustre --mgsnode=192.168.100.150@o2ib /dev/sda2 # lctl get_param lquota.*.quota_type lquota.lustre-MDT0000.quota_type=2 lquota.lustre-OST0000.quota_type=1 lquota.lustre-OST0001.quota_type=3 lquota.lustre-OST0002.quota_type=3 </pre> </div></div> <p>Set ost.quota_type=ug3 and confirmed it's set to ug3 correctly</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># tunefs.lustre --erase-param /dev/sda2 # tunefs.lustre --mgsnode=192.168.100.150@o2ib --param=ost.quota_type=ug3 /dev/sda2 # lctl get_param lquota.*.quota_type lquota.lustre-MDT0000.quota_type=2 lquota.lustre-OST0000.quota_type=3 lquota.lustre-OST0001.quota_type=3 lquota.lustre-OST0002.quota_type=3 </pre> </div></div> <p>MDS was no problem.</p> <p>If we set ug2 to MDT, it's running with ug2 and if we set ug3, it works with ug3 as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkfs.lustre --reformat --mgs --mdt /dev/sda1 # mount -t lustre /dev/sda1 /mnt/lustre/sda1 # lctl get_param lquota.*.quota_type lquota.lustre-MDT0000.quota_type=3 </pre> </div></div> <p>Thanks, Ihara</p> <p>The quota_type of OST is 1 (which means 32bit quota limit on OSTs), so 160T has exceeded the limit range, it's not supported.</p> <p>Following is the comment for quuota_type:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> * MDS: u <span class="code-keyword">for</span> user quotas (administrative+operational) turned on, * g <span class="code-keyword">for</span> group quotas (administrative+operational) turned on, * 1 <span class="code-keyword">for</span> 32-bit operational quotas and 32-bit administrative quotas, * 2 <span class="code-keyword">for</span> 32-bit operational quotas and 64-bit administrative quotas, * 3 <span class="code-keyword">for</span> 64-bit operational quotas and 64-bit administrative quotas * OST: u <span class="code-keyword">for</span> user quotas (operational) turned on, * g <span class="code-keyword">for</span> group quotas (operational) turned on, * 1 <span class="code-keyword">for</span> 32-bit local operational quotas, * 2 <span class="code-keyword">for</span> 32-bit local operational quotas, * 3 <span class="code-keyword">for</span> 64-bit local operational quotas, </pre> </div></div> <p>So, if you want 64bit local quota limit, you should specifiy the quota_type as 3. No matter if you specify 1 or 2 to OST quota_type, OST will use 32bit quota limit, and the proc file always dispaly 1 for OSTs.</p> <p>If you want change your current MDS/OSTs to 64bit quota limit, you need to specify 3 for both MDS &amp; OSTs, then run 'lfs quotacheck' to regenerate the quota files. (I suppose your kernel support 64 bit quota limit).</p> <p>Thanks Niu for confirmation.</p> <p>So, what is "operations qutoas" and "administrative quotas", btw?</p> <p>4TB is quota limitation per OST if quota if quota_type is 1 or 2.<br/> The total 42 OSTs are running on here, 168TB is user and group quota limitation on the entire filesystem.<br/> We have been seeing this problem once a group usage per OST is going to close of 4TB and achieved.</p> <p>To enable 64bit quota on the production system, can the following changes work without restart?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lctl set_param lquota.*.quota_type=3 # lfs quotacheck &lt;fs&gt; </pre> </div></div> <p>For permanent fix, need to rewrite ost.quota_type=ug3 to OSTs with tunfs.lustre if we have an chance to stop the OSTs.</p> <blockquote> <p>So, what is "operations qutoas" and "administrative quotas", btw?</p></blockquote> <p>Operational quotas is the quota file of the local fs for each OST/MDS, which is used for storing usage &amp; local limit for each target. Administrative quotas is the cluster wide quota file, which is used to store quota limit (clustre wide), the administrative quota file is located on MDS.</p> <blockquote> <p>To enable 64bit quota on the production system, can the following changes work without restart?</p></blockquote> <p>Yes. Actually, you can use "lctl conf_param $MDTNAME.mdt.quota_type=ug3" &amp; "lctl conf_param $OSTNAME.ost.quota_type=ug3" to write it in the config log permanently.</p> <blockquote> <p>Operational quotas is the quota file of the local fs for each OST/MDS, which is used for storing usage &amp; local limit for each target. Administrative quotas is the cluster wide quota file, which is used to store quota limit (clustre wide), the administrative quota file is located on MDS.</p></blockquote> <p>OK, that makes perfect sense.</p> <blockquote> <p>Yes. Actually, you can use "lctl conf_param $MDTNAME.mdt.quota_type=ug3" &amp; "lctl conf_param $OSTNAME.ost.quota_type=ug3" to write it in the config log permanently.</p></blockquote> <p>Yes, that also worked on my test system and I've confirmed quota_type is set to ug3 permanently. if we see params by tunefs.lustre, the params of OST is not updated though (still ug2). sometimes confused. anyway, conf_param can be fixed.</p> <p>Niu,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lctl set_param lquota.*.quota_type=3 # lfs quotacheck &lt;fs&gt; </pre> </div></div> <p>It doesn't help to convert from ug2 to ug3. "lctl conf_param $MDTNAME.mdt.quota_type=ug3" &amp; "lctl conf_param $OSTNAME.ost.quota_type=ug3" and restart OST/MDT, doesn't help either. We still got quota exceeded when an user writes 4TB data to single OST even quota_type=ug3 is set.</p> <p>In order to convert from v2 to v3 and support 64bit completely, it needed umount MDT/OSTs and mount them with ldiskfs, and remove lquota.</p> {user,group} and lquota_v2.{user,group} <p> files, then restart all MDT/OST and 'lfs quotacheck', again...</p> <p>hmm, set the quota_type as "ug3" then run 'lfs quotacheck" doesn't help?</p> <p>"lctl set_param lquota.<b>.quota_type=3" isn't correct, because it doesn't enable user or group quota, quotacheck will not recreate any quota files. I think it should be "lctl set_param lquota.</b>.quota_type=ug3", then run quotacheck. Sorry I didn't find this trap before.</p> <p>Thanks for your effort on this, and I'm glad to hear that you've convert the quota files successfully.</p> <p>Originally, this is one of configuration problem, but finally, the root cause of this problem was <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a>. So, please close this ticket, anyway. Thanks!</p> <p>ok thanks Ihara!</p> Development Rank 1|hzvgsf: Rank (Obsolete) 6369 Severity