https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1438 Lustre <p>we are getting some of quota related problem. the quota feature is enabled on the filesystem and the customer changed group name to the many big files, then ran lfs quotacheck command.<br/> After that, even the group didn't exceed the quota limitation, they got the disk quota exceeded messages.</p> <p>OSS/MDS side, the following messages showed up since changed group name.<br/> (quota_interface.c:473:quota_chk_acq_common()) still haven't managed to<br/> acquire quota space from the quota master after 20 retries (err=0, rc=0)</p> <p>It seems to be close to <a href="https://jira.whamcloud.com/browse/LU-428" title="Lustre: 16290:0:(quota_interface.c:460:quota_chk_acq_common()) still haven&#39;t managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)" class="issue-link" data-issue-key="LU-428"><del>LU-428</del></a>.</p> lustre-1.8.7-wc1, RHEL5.7 for servers, RHEL6.2 for clients LU-1438

quota_chk_acq_common() still haven't managed to acquire quota

Bug Minor Resolved Duplicate Niu Yawei Shuichi Ihara Thu, 24 May 2012 05:54:56 +0000 Fri, 22 Feb 2013 11:14:38 +0000 Sat, 18 Aug 2012 10:34:38 +0000 Lustre 1.8.7 Lustre 2.3.0 Lustre 2.1.4 Lustre 1.8.9 0 5 <p>Niu</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Yes, it looks similar to the <a href="https://jira.whamcloud.com/browse/LU-428" title="Lustre: 16290:0:(quota_interface.c:460:quota_chk_acq_common()) still haven&#39;t managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)" class="issue-link" data-issue-key="LU-428"><del>LU-428</del></a>, but we can't reproduced it in <a href="https://jira.whamcloud.com/browse/LU-428" title="Lustre: 16290:0:(quota_interface.c:460:quota_chk_acq_common()) still haven&#39;t managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)" class="issue-link" data-issue-key="LU-428"><del>LU-428</del></a> unfortunately.</p> <p>Ihara, if it's reproduceable, could you try to collect the debug log on both MDS &amp; OST with D_QUOTA enabled? Thanks.</p> <p>Did customer run 'lfs setquota' to clear limit for some user/group when they see this problem? Is it easy to be reproduced? I found a race condition in the code which could lead to such situation, but I'm not sure if it's the real root cause. Anyway, I'll post a patch to fix that race.</p> <p>Niu,<br/> Let me ask the customer, even if they clear the quota by "lfs set quota", they see same issue.<br/> should we test this first without debug?</p> <p>Hi, Ihara</p> <p>The race I mentioned is hard to be triggerred, so we'd better get the debug log if it can be reproduced.</p> <p>The patch to fix race in quota_chk_acq_common(): <a href="http://review.whamcloud.com/#change,2927" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2927</a></p> <p>Niu,</p> <p>I upaded the debug information when debug flags turned on. <br/> please check on /uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/20120529.tar.bz2 on the whamcloud's ftp site.</p> <p>Thanks<br/> Ihara</p> <p>Thank you, Ihara. There are lots of "still haven't managed to acquire quota.." in the messages, however, seems the debug log was truncated, I can't find any of them in the debug log.</p> <p>Could you try the patch above to see if it resolves cursomer's problem?</p> <p>Niu,<br/> ok, let me try patch to see if the problem is fixed. <br/> Is this needed to apply to both servers and clients?</p> <p>This is a server patch, you can apply it on server only.</p> <p>Is this fix needed on master too?</p> <p>Hi, Peter, yes, I'll cook a patch for master.</p> <p>patch for master: <a href="http://review.whamcloud.com/#change,2996" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2996</a></p> <p>Niu,<br/> Thanks, we are applying patches.<br/> what exactly problem is? what's situation why we enter infinite loop?</p> <p>This is a race in the quota code, which could be triggered when user execute 'lfs setquota' to clear limit for some user/group on an active system (quota enabled, and there is ongoing io).</p> <p>The customer did "lfs setquota", but it was not to clear. they changed quota limit (exceed original quota limit, then up the quota size) for some of user/group. After that, we have been seeing the these messages.<br/> So, we didn't clear quota, but during the changing operation for quota size, the quota size is cleared once, then set new quota size?</p> <p>The limit size changing will not cause clear -&gt; set operations. So it maybe not caused by this race, anyway, let's see the result frist.</p> <p>so, what do you excactly mean "lfs setquota" to clear, is this setting to 0 for all quota limits for user/group?</p> <p>Niu,</p> <p>We only applied patches to an OSS of all of them, but it didn't help. still messages are showing up.</p> <p>Please investigate further and let me if you need more information. </p> <p>uploaded patch applied OSS's /var/log/messages on ftp site. /uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/messages.nos14.tgz<br/> Please have a look at it.</p> <p>Hi, Ihara</p> <p>This time the warnning message is little bit different:</p> <p>Jun 5 13:27:30 nos141i kernel: Lustre: 28834:0:(quota_interface.c:473:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 51 retries (err=0, rc=-5)</p> <p>The return value is -5 now, which means some error happened during ost acquiring quota from mds. In the previous messages, the rc is always 0.</p> <p>Can this OSS work properly even without quota enabled?</p> <p>Niu, </p> <p>Yes, quota has been turned off on all servers, but still showing this messages on the OSS.</p> <p>btw, we only applied patch to an OSS, but not to MDS yet.</p> <p>Niu, </p> <p>the following error messages is showing up on MDS forever..</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 9831:0:(ldlm_lib.c:2123:target_handle_dqacq_callback()) dqacq failed! (rc:-5) </pre> </div></div> <p>posted MDS's /var/log/messages as /uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/nmd05.tgz</p> <p>hmm, I'm afraid that the quota isn't turned off on the OSS, could you check the /proc/fs/lustre/obdfilter/$OSTNAME/quota_type to see if it's really turned off?</p> <p>Apply the patch to one OSS is fine.</p> <p>some storage here.. it indicates "quota is disabled" on the client, and nothing "u" and "g" on MDS and OSS, but quota_type is set to 1 on OSSs, but MDS's quota_type is 2.</p> <p>&#35; lfs quota -u -t /nshare1<br/> user quotas are not enabled.<br/> &#35; lfs quota -g -t /nshare1<br/> group quotas are not enabled.</p> <p>on OSSs<br/> &#35; lctl get_param obdfilter.*.quota_type<br/> obdfilter.nshare1-OST0015.quota_type=1<br/> obdfilter.nshare1-OST0016.quota_type=1<br/> obdfilter.nshare1-OST0017.quota_type=1<br/> obdfilter.nshare1-OST0018.quota_type=1<br/> obdfilter.nshare1-OST0019.quota_type=1<br/> obdfilter.nshare1-OST001a.quota_type=1<br/> obdfilter.nshare1-OST001b.quota_type=1</p> <p>on MDS<br/> &#35; cat /proc/fs/lustre/lquota/nshare3-MDT0000/quota_type <br/> 2</p> <p>Niu, </p> <p>any advices? it still can't stop error messages on MDS.. the customer is nervous about this..</p> <p>Ihara</p> <p>Hi, Ihara</p> <p>The 'quota_type' on both MDS &amp; OSSs are looks good, seems no user or group quota is enabled. I don't where the dqacq requests come from, I think we have to enable D_QUOTA on both MDS &amp; OSSs, and collect some debug log for analysis. Thanks.</p> <p>Niu,</p> <p>we collected debug logs on MDS and two OSSs and filed them on /uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/2012-06-07/<br/> We had been getting debug log until the following error messages on OSS.<br/> kernel: Lustre: 23619:0:(quota_interface.c:473:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 78 retries (err=67, rc=-5)</p> <p>Please investigate futher.</p> <p>Thank you very much!</p> <p>Thank you, Ihara. Seems it's a race of turn off quota vs. acquire quota, I will post a patch soon.</p> <p>patch for b18: <a href="http://review.whamcloud.com/3060" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3060</a></p> <p>Thank you very much for this patch and we will try this patch as well.<br/> This problem is that the quota slave is infinitely acquiring quota even quota is turned off on master. Because there is missing of check if quota is enabled, is it correct?<br/> So, we should apply this patch on both MDS and OSS, right?</p> <p>yes, exactly. The change is only for slave, but it should be applied on both MDS &amp; OSS, since MDS has a slave running on it as well.</p> <p>Hi, Ihara</p> <p>After looking closer to the log, I realized that your case is not caused by the race, it probably caused by that when you disable quota, some OST is offline (or the clear quota command to some OST was failed for some reason), so the local fs quota isn't disabled on some OST.</p> <p>To resolve the problem, you can return the 'lfs quotaoff' to see if it's helpful. Thanks.</p> <p>Niu,</p> <p>Thanks for updates. the original problem was that some groups exceeded quota despite these groups didn't achieve quota limit. what do you think of this problem?</p> <p>The following messages showed up after disabled quota, but the primary problem was quota limit didn't work well.<br/> kernel: Lustre: 23619:0:(quota_interface.c:473:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 78 retries (err=67, rc=-5)</p> <blockquote> <p>the original problem was that some groups exceeded quota despite these groups didn't achieve quota limit. what do you think of this problem?</p></blockquote> <p>The original problem is that slave can't acquire quota from master for that group id, so it exceeded quota. My first patch might be helpful for this. You can also ask user to reset the quota limit for the problematic group id by 'lfs setquota 0' -&gt; 'lfs setquota xxx' to see if it's helpful.</p> <blockquote> <p>The following messages showed up after disabled quota, but the primary problem was quota limit didn't work well.<br/> kernel: Lustre: 23619:0:(quota_interface.c:473:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 78 retries (err=67, rc=-5)</p></blockquote> <p>The second problem might be caused by the inconsistence of on/off information between slave and master, you could ask user to rerun the 'lfs quotaoff' to see if it works.</p> <p>Anyway, the quota design (in master &amp; b18) requires all slaves online while setting quota or turn on/off quota, if the setquota or enable/disable quota failed on some slave (OST), there will be trouble, you have to rerun the command to make sure the new setting is synchronized on each slave(OST). Such kind of design drawback has been addressed in the new quota design (in orion).</p> <p>Niu,</p> <p>It seems that the patches didn't help fixing problem.. <br/> we stopped all OSSs and MDSs and restarted the Lustre with applied RPMs.<br/> THe applied patches are below.<br/> <a href="http://review.whamcloud.com/#change,3060" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3060</a><br/> <a href="http://review.whamcloud.com/#change,2927" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2927</a></p> <p>However, a user got "quota exceeded messages" even he didn't get quota limit.<br/> At that time, we got the following messages on the OSS.</p> <p>Jun 12 18:19:36 nos131i kernel: Lustre: 29689:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)<br/> Jun 12 18:19:36 nos131i kernel: Lustre: 29689:0:(quota_interface.c:481:quota_chk_acq_common()) Skipped 10 previous similar messages</p> <p>And not same time, but we are seeing a lot of same messages on the even another OSSs.</p> <p>The resent logfiles are filed on /uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/20120613.</p> <p>Please investigate further. thank you very much again.</p> <p>patch for master: <a href="http://review.whamcloud.com/#change,3097" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3097</a></p> <p>Hi, Ihara</p> <p>Did you try reset quota by 'lfs setquota' for the user?</p> <p>You mean set zero to all (e.g. "lfs setquota -u &lt;user&gt; -b 0 -B 0 -i 0 -I 0 &lt;fsname&gt;") to clear, then set real quota size again?</p> <p>Yes, for the user who has quota limit, reset quota by "lfs setquota -b 0 -B 0 -i 0 -I 0 $fsname" -&gt; "lfs setquota -b limit ...", for the user who has no quota limit but got the error message (still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)) on OSS, re-clear the quota limit by 'lfs setquota -b limit ..." -&gt; 'lfs setquota -b 0 ...".</p> <p>There isn't much information in the dmesg, could you explain current status of the system? is user or group quota enabled? what's the output of 'lfs quota -v -u $user $fs" for the user who has no quota limit but got error message on OSS? Thanks.</p> <p>Niu,</p> <p>The customer did "lfs setquota" to clear the all quota size to some users.<br/> After that, as far as we can see, messages "(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)" are gone so far.</p> <p>As you mentioned, there was an inconsistency between master and slave - user is not exceeded to the global limit, but some local quota were exceeded the limit. Once the local quota on all MDS/OSSs are cleared by "lfs setquota -b 0 -B 0...", then set quota again, now the quota works normally. Is this what you said a root cause and solution to fix current situation, right?</p> <p>So, for change the quota size, "1. clear quota size, 2. set quota size" is better and safe solution rather than just change the quota size by "lfs setquota -b X"?</p> <p>Anyway, very appreciate you detailed analysis and explanation.</p> <p>To change quota size, "clear -&gt; set new size" isn't necessary safer than 'set new size' directly.</p> <p>Whenever 'lfs setquota' changes quota limit from zero to non-zero (or from non-zero to zero), quota master (MDS) will notify all slaves (OSTs) to change their local fs quota limit, however, if some slave is offline at that time (or the notification to some slave failed for some reason), inconsistence will raise between the master and the slaves which didn't receive quota change notification.</p> <p>Changing quota from old limit to new limit (all non-zero values) will not trigger the quota change notification to slaves, so it'll not cause the inconsistence, of course, it can't fix the inconsistence as well.</p> <p>In the new quota design, any later joined slave can sync the quota setting with master automatically, so user needn't worry about the offline slave anymore.</p> <p>Hello Niu,</p> <p>The problem is not fixed yet even after reset group quota. The last week, we did zero reset quota for all groups, but a couple of groups were failed to set quota above. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>nos13: Jun 20 18:02:19 nos131i kernel: Lustre: 18627:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) nos13: Jun 20 21:40:47 nos131i kernel: Lustre: 29660:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) nos14: Jun 18 13:10:06 nos141i kernel: LustreError: 18554:0:(quota_ctl.c:260:filter_quota_ctl()) fail to create lqs during setquota operation for gid 10856 nos14: Jun 18 13:10:06 nos141i kernel: LustreError: 18575:0:(quota_ctl.c:260:filter_quota_ctl()) fail to create lqs during setquota operation for gid 10857 nos14: Jun 20 12:38:41 nos141i kernel: Lustre: 28944:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) </pre> </div></div> <p>Then, we set quota again to these groups, but today, we got same messages on OSS.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 25 03:03:59 nos141i kernel: Lustre: 28760:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) </pre> </div></div> <p>I don't know why still failing to acquire quota space from master. Any ideas of workaround to avoid this issue? or we can add debug to address this issue and to understand what happens?</p> <p>Hi, Ihara</p> <p>If it's easy to be reproduced, we can collect the debug log on OSS with D_TRACE &amp; D_QUOTA enabled (echo +trace &gt; /proc/sys/lnet/debug, echo +quota &gt; /proc/sys/lnet/debug), then we can see where the acquire quota returns zero. Thanks.</p> <p>Niu, I meant you could make debug patch to see more detail information during the production system. e.g) Who exceed quota limit by this messages. surely, we didn't do any operations (set/clear quota) when the following messages showed up. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 25 03:03:59 nos141i kernel: Lustre: 28760:0:(quota_interface.c:481:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) </pre> </div></div> <p>We don't know how reproduce this problem at this morment, but let me ask we can enable debug flags. BTW, what is "28760:0" of above messages?</p> <p>Hi, Ihara</p> <p>The debug patch would be similar to enable D_TRACE &amp; D_QUOTA for debug log, If the cusotmer can't affort D_TRACE debug log, we can only enable D_QUOTA first to collect some debug log.</p> <p>The 28760 is pid, and the 0 is 'extern pid' (looks it's always 0 for now, you can just ignore it).</p> <p>Hi Niu,</p> <p>uploaded debug files on uploads/<a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>/debugfile.20120628.gz<br/> However, it's not easy to reproduce this messages though the messages still showed up on the system log file irregularly.<br/> The lustre debug file doesn't contain messages. perhaps, the maximum debug size (100MB) exceeded quickly?<br/> Any ideas to keep track and debug information in this situation? </p> <p>Hi, Ihara</p> <p>Could you apply this debug patch? then we'll see lots more deubug information in the syslong along with the "still can't acquire..." messages. Thanks.</p> <p>Hi Niu,</p> <p>Thanks for this debug patch. I will ask customer if we can apply patch.</p> <p>This might be related to 32bit quota setting limitation?</p> <p>probably, is there anything useful from the debug log?</p> <p>We didn't apply patches yet, but will apply them on the next scheduled maintenance time.</p> <p>this might be caused by <a href="https://jira.whamcloud.com/browse/LU-1584" title="error set quota fs limit" class="issue-link" data-issue-key="LU-1584"><del>LU-1584</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a>. Please close this ticket. <br/> I will open new ticket if we see same issue after apply <a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a> patches.</p> <p>ok thanks Ihara</p> Development Rank 1|hzv6hb: Rank (Obsolete) 4584 Severity