https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2289 Lustre <p>Even after adding the patch from <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><del>LU-1720</del></a>, we are still seeing messages like:<br/> Lustre: 18271:0:(quota_interface.c:475:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) </p> <p>At this site, we haven't added any other patches to 1.8.8. What do these messages mean? Is it possible that some of the other patches (like <a href="https://jira.whamcloud.com/browse/LU-1438" title="quota_chk_acq_common() still haven&#39;t managed to acquire quota" class="issue-link" data-issue-key="LU-1438"><del>LU-1438</del></a>) could fix these?</p> <p>The customer hasn't noticed any functional issues, but of course that doesn't mean there aren't any. Quotas &gt;4TB work on this system.</p> Lustre 1.8.8 + <a href="https://jira.whamcloud.com/browse/LU-1720" title="Quota doesn&#39;t work over 4TB on single OST" class="issue-link" data-issue-key="LU-1720"><strike>LU-1720</strike></a> LU-2289

still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)

Bug Minor Resolved Not a Bug Niu Yawei Oz Rentas ptr Tue, 6 Nov 2012 12:05:31 +0000 Tue, 29 Sep 2015 08:22:52 +0000 Tue, 29 Sep 2015 08:22:52 +0000 Lustre 1.8.8 0 6 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I see lots of following errors in mds log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jul 12 20:34:19 lfs-mds-2-1 kernel: Lustre: 7972:0:(lproc_quota.c:453:lprocfs_quota_wr_type()) lfs2-MDT0000: quotaon failed because quota files don't exist, please run quotacheck firstly </pre> </div></div> <p>Seems there is something wrong on MDT, so that quota was not turned on for MDT, but it's turned on for OST, so OST can't acquire quota from MDT and report the messages "still haven't managed to acquire quota space from the quota master after 10 retries".</p> <p>Could you check what's the quota configuration was used? (ug3 for both MDT &amp; OST?) If rerun "lfs quotacheck" can fix your problem? I not, I think we'd collect debuglog for quotacheck to see why the amdin or local quota file wasn't created. Thanks.</p> <p>Hi Niu,</p> <p>Sorry I didn't truncate the logs to the relevant dates, those are actually old messages. I will reupload the logs with only the relevant parts. </p> <p>In preparing these logs, I noticed that when the OSS was displaying the message, the MDT was having problems connecting:<br/> Aug 30 22:28:11 lfs-mds-2-1 kernel: Lustre: 9067:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1411737505069379 sent from lfs2-OST0002-osc to NID 10.179.16.124@o2ib 0s ago has failed due to network error (15s prior to deadline).</p> <p>This actually looks like:<br/> <a href="http://jira.whamcloud.com/browse/LU-1809" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1809</a></p> <p>I will try to get more information from the customer about when they are seeing the message and if they are running with the workaround suggested in <a href="https://jira.whamcloud.com/browse/LU-1809" title="Clients unable to mount (-108)" class="issue-link" data-issue-key="LU-1809"><del>LU-1809</del></a>. </p> <p>Thanks,<br/> Kit</p> <p>logs only for Aug 30</p> <p>Thank you, Kit.</p> <p>Yes, the new log shows quite a lot of connection errors, I'm not sure if the filesystem was healty at that time.</p> <p>The most possible reason of OST constantly showing "still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)" is that there are some kind of inconsistency between master (MDT) and slave (OST):</p> <ul class="alternate" type="square"> <li>Quota is already disabled on master (MDT), but not disabled on slave (OST);</li> <li>Quota limit for some certain uid/gid has been cleared on MDT (master), but not cleared on slave (OST);</li> </ul> <p>If the quota is supposed to be disabled, but writting files can cause the "still haven't...(err=0, rc=0)" message on OSS, you can rerun "lfs quotaoff" when all OSTs are up to make sure quota is disabled on each OST;<br/> If some uid/gid is supposed to have 0 limit, but writting to the uid/gid owned files can cause the message on OSS, you can rerun "lfs setquota ..." when all OSTs are up to make sure the limit is cleared on each OST;</p> <p>That's what I can think of so far.</p> <p>We've seen this issue again without any network issues. Is there any way to debug what is going on? The error only appears intermittently, so I think it's probably some kind of sync issue. Is there a way to tell what uid/gid is causing problems, or is it better to just redo the lfs quotacheck? </p> <p>Hi Niu,</p> <p>The customer ran lfs quotaoff, quotaon, and quotacheck. The problem reappeared fairly quickly on one filesystem, and after a few weeks to the other filesystem. </p> <p>The customer asks:<br/> What is the impact of NOT acquiring quota space?</p> <p>I noticed that the quota debug level prints information about the UID/GID that is causing problems. I will try to get quota debug logs during the event. Is there anything else we can get, or any other ideas?</p> <p>Thanks,<br/> Kit</p> <p>If the quota acquire constantly fail like this, the io will not finished.</p> <p>Could you run "lfs quota -v uid/gid" to see what's the quota limits of the uid/gid which is causing the problem? Thanks.</p> <p>Hi Niu,</p> <p>I was able to run with quota debug enabled. The debug logs are very large, but I think we were able to see which groups were having the logs messages printed. There doesn't appear to be anything strange about them:</p> <p>Disk quotas for group hfip-gfdl (gid 19103):<br/> Filesystem kbytes quota limit grace files quota<br/> limit grace<br/> /lfs1 18016033720 20971520000 22020096000 - 3544247 0 0 -</p> <p>Would it be possible to see whether or not the quota space was eventually acquired? Do you think I would be able to trace it by pid? Or do different threads operate on the same quota task? Is there a transaction ID or anything? I'll look at the code too to see if I can tell. </p> <p>Also I could make the logs available. They are each about ~300MB compressed, ~11GB uncompressed. </p> <p>Thanks</p> <p>Here are the quotas with -v:</p> <p><span class="error">&#91;root@Jet:fe6 ~&#93;</span># lfs quota -v -g hfipprd /lfs1<br/> Disk quotas for group hfipprd (gid 10074):<br/> Filesystem kbytes quota limit grace files quota limit<br/> grace<br/> /lfs1 0 31457280 33030144 - 0 0<br/> 0 -<br/> lfs1-MDT0000_UUID<br/> 0 - 65536 - 0 - 0<br/> -<br/> lfs1-OST0000_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0001_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0002_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0003_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0004_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0005_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0006_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0007_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0008_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0009_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0010_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0011_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0012_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0013_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0014_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0015_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0016_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0017_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0018_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0019_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0020_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0021_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0022_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0023_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0024_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0025_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0026_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0027_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0028_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0029_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0030_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0031_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0032_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0033_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0034_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0035_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0036_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0037_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0038_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0039_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0040_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0041_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0042_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0043_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0044_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0045_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0046_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0047_UUID<br/> 0 - 65536 - - - -<br/> -</p> <p><span class="error">&#91;root@Jet:fe6 ~&#93;</span># lfs quota -v -g hfip-gfdl /lfs1<br/> Disk quotas for group hfip-gfdl (gid 19103):<br/> Filesystem kbytes quota limit grace files quota limit<br/> grace<br/> /lfs1 19179828108 20971520000 22020096000 - 3567465<br/> 0 0 -<br/> lfs1-MDT0000_UUID<br/> 1408244 - 1572864 - 3567465 - 0<br/> -<br/> lfs1-OST0000_UUID<br/> 266307944 - 266469376 - - -</p> <ul class="alternate" type="square"> <li>-<br/> lfs1-OST0001_UUID<br/> 264490536 - 264634368 - - -</li> <li>-<br/> lfs1-OST0002_UUID<br/> 252475224 - 252575744 - - -</li> <li>-<br/> lfs1-OST0003_UUID<br/> 267990876 - 268173312 - - -</li> <li>-<br/> lfs1-OST0004_UUID<br/> 246162144 - 246284288 - - -</li> <li>-<br/> lfs1-OST0005_UUID<br/> 267940220 - 268042240 - - -</li> <li>-<br/> lfs1-OST0006_UUID<br/> 290532840 - 290717696 - - -</li> <li>-<br/> lfs1-OST0007_UUID<br/> 283466396 - 283639808 - - -</li> <li>-<br/> lfs1-OST0008_UUID<br/> 272318500 - 272498688 - - -</li> <li>-<br/> lfs1-OST0009_UUID<br/> 250366252 - 250478592 - - -</li> <li>-<br/> lfs1-OST000a_UUID<br/> 268765900 - 268959744 - - -</li> <li>-<br/> lfs1-OST000b_UUID<br/> 241840752 - 241958912 - - -</li> <li>-<br/> lfs1-OST000c_UUID<br/> 279475784 - 279576576 - - -</li> <li>-<br/> lfs1-OST000d_UUID<br/> 295138616 - 295305216 - - -</li> <li>-<br/> lfs1-OST000e_UUID<br/> 245769816 - 245891072 - - -</li> <li>-<br/> lfs1-OST000f_UUID<br/> 258449640 - 258605056 - - -</li> <li>-<br/> lfs1-OST0010_UUID<br/> 228665128 - 228851712 - - -</li> <li>-<br/> lfs1-OST0011_UUID<br/> 278386580 - 278528000 - - -</li> <li>-<br/> lfs1-OST0012_UUID<br/> 277253652 - 277348352 - - -</li> <li>-<br/> lfs1-OST0013_UUID<br/> 185330656 - 185466880 - - -</li> <li>-<br/> lfs1-OST0014_UUID<br/> 281571296 - 281673728 - - -</li> <li>-<br/> lfs1-OST0015_UUID<br/> 246132636 - 246284288 - - -</li> <li>-<br/> lfs1-OST0016_UUID<br/> 274774304 - 274857984 - - -</li> <li>-<br/> lfs1-OST0017_UUID<br/> 279671348 - 279838720 - - -</li> <li>-<br/> lfs1-OST0018_UUID<br/> 278511144 - 278659072 - - -</li> <li>-<br/> lfs1-OST0019_UUID<br/> 279721016 - 279838720 - - -</li> <li>-<br/> lfs1-OST001a_UUID<br/> 286078396 - 286261248 - - -</li> <li>-<br/> lfs1-OST001b_UUID<br/> 284810104 - 284950528 - - -</li> <li>-<br/> lfs1-OST001c_UUID<br/> 266730624 - 266862592 - - -</li> <li>-<br/> lfs1-OST001d_UUID<br/> 267699528 - 267780096 - - -</li> <li>-<br/> lfs1-OST001e_UUID<br/> 271926964 - 272105472 - - -</li> <li>-<br/> lfs1-OST001f_UUID<br/> 279551644 - 279707648 - - -</li> <li>-<br/> lfs1-OST0020_UUID<br/> 262360628 - 262537216 - - -</li> <li>-<br/> lfs1-OST0021_UUID<br/> 286415060 - 286523392 - - -</li> <li>-<br/> lfs1-OST0022_UUID<br/> 227815636 - 227934208 - - -</li> <li>-<br/> lfs1-OST0023_UUID<br/> 271352884 - 271450112 - - -</li> <li>-<br/> lfs1-OST0024_UUID<br/> 272460280 - 272629760 - - -</li> <li>-<br/> lfs1-OST0025_UUID<br/> 198046512 - 198180864 - - -</li> <li>-<br/> lfs1-OST0026_UUID<br/> 289555288 - 289669120 - - -</li> <li>-<br/> lfs1-OST0027_UUID<br/> 258436324 - 258605056 - - -</li> <li>-<br/> lfs1-OST0028_UUID<br/> 270941900 - 271056896 - - -</li> <li>-<br/> lfs1-OST0029_UUID<br/> 251377320 - 251527168 - - -</li> <li>-<br/> lfs1-OST002a_UUID<br/> 282350136 - 282460160 - - -</li> <li>-<br/> lfs1-OST002b_UUID<br/> 280608276 - 280756224 - - -</li> <li>-<br/> lfs1-OST002c_UUID<br/> 284707076 - 284819456 - - -</li> <li>-<br/> lfs1-OST002d_UUID<br/> 283481900 - 283639808 - - -</li> <li>-<br/> lfs1-OST002e_UUID<br/> 283875928 - 284033024 - - -</li> <li>-<br/> lfs1-OST002f_UUID<br/> 241472472 - 241565696 - - -</li> <li>-<br/> lfs1-OST0030_UUID<br/> 265680488 - 265814016 - - -</li> <li>-<br/> lfs1-OST0031_UUID<br/> 273082788 - 273154048 - - -</li> <li>-<br/> lfs1-OST0032_UUID<br/> 227469276 - 227540992 - - -</li> <li>-<br/> lfs1-OST0033_UUID<br/> 301283952 - 301465600 - - -</li> <li>-<br/> lfs1-OST0034_UUID<br/> 282200664 - 282329088 - - -</li> <li>-<br/> lfs1-OST0035_UUID<br/> 236723324 - 236847104 - - -</li> <li>-<br/> lfs1-OST0036_UUID<br/> 244762536 - 244842496 - - -</li> <li>-<br/> lfs1-OST0037_UUID<br/> 290396060 - 290586624 - - -</li> <li>-<br/> lfs1-OST0038_UUID<br/> 247838916 - 247988224 - - -</li> <li>-<br/> lfs1-OST0039_UUID<br/> 289519172 - 289669120 - - -</li> <li>-<br/> lfs1-OST003a_UUID<br/> 237240432 - 237371392 - - -</li> <li>-<br/> lfs1-OST003b_UUID<br/> 281243232 - 281411584 - - -</li> <li>-<br/> lfs1-OST003c_UUID<br/> 255128092 - 255197184 - - -</li> <li>-<br/> lfs1-OST003d_UUID<br/> 267801692 - 267911168 - - -</li> <li>-<br/> lfs1-OST003e_UUID<br/> 268842936 - 268959744 - - -</li> <li>-<br/> lfs1-OST003f_UUID<br/> 259531064 - 259653632 - - -</li> <li>-<br/> lfs1-OST0040_UUID<br/> 270812788 - 270925824 - - -</li> <li>-<br/> lfs1-OST0041_UUID<br/> 256516360 - 256638976 - - -</li> <li>-<br/> lfs1-OST0042_UUID<br/> 289071672 - 289144832 - - -</li> <li>-<br/> lfs1-OST0043_UUID<br/> 290809280 - 290979840 - - -</li> <li>-<br/> lfs1-OST0044_UUID<br/> 265673448 - 265814016 - - -</li> <li>-<br/> lfs1-OST0045_UUID<br/> 288587388 - 288751616 - - -</li> <li>-<br/> lfs1-OST0046_UUID<br/> 291939312 - 292028416 - - -</li> <li>-<br/> lfs1-OST0047_UUID<br/> 232700912 - 232783872 - - -</li> <li>-</li> </ul> <p><span class="error">&#91;root@Jet:fe6 ~&#93;</span># lfs quota -v -g rthfip-gfdl /lfs1<br/> Disk quotas for group rthfip-gfdl (gid 19104):<br/> Filesystem kbytes quota limit grace files quota limit<br/> grace<br/> /lfs1 0 31457280 33030144 - 0 0<br/> 0 -<br/> lfs1-MDT0000_UUID<br/> 0 - 65536 - 0 - 0<br/> -<br/> lfs1-OST0000_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0001_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0002_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0003_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0004_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0005_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0006_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0007_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0008_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0009_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST000f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0010_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0011_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0012_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0013_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0014_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0015_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0016_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0017_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0018_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0019_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST001f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0020_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0021_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0022_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0023_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0024_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0025_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0026_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0027_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0028_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0029_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST002f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0030_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0031_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0032_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0033_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0034_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0035_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0036_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0037_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0038_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0039_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003a_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003b_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003c_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003d_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003e_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST003f_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0040_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0041_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0042_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0043_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0044_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0045_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0046_UUID<br/> 0 - 65536 - - - -<br/> -<br/> lfs1-OST0047_UUID<br/> 0 - 65536 - - - -<br/> -</p> <p>Hi, Kit</p> <p>If you have debug log, you can see if the quota is acquired eventually by checking the pid. (find the pid in the message of "still haven't managed ...", and check if this thread is constantly printing such message or it moves on eventually).</p> <p>When you attach the log, I suggest you truncate out the useful part, otherwise, it could be too large. The useful part should be the messages nearby "still haven't managed ..." OST and the corresponding messages on MDS.</p> <p>I didn't find anything abnormal in the output of "lfs quota -u -v", thank you for providing this.</p> <p>Hi Niu,</p> <p>I've uploaded the logs here:<br/> <a href="http://eu.ddn.com:8080/lustre/LU-2289.tar.bz2" class="external-link" target="_blank" rel="nofollow noopener">http://eu.ddn.com:8080/lustre/LU-2289.tar.bz2</a></p> <p>It's about a 3 second window on each server, but it's still about 20GB uncompressed. It looks like for the message at 1364224183, it tries 19 times before it finally acquires it. Is there anything in the MDT logs that would indicate why it took so long? I didn't see anything in particular, but it's tough to read.</p> <p>Thank you, Kit. I checked the OSS log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00040000:00000400:6:1364224183.636169:0:1558:0:(quota_interface.c:475:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0) 00040000:04000000:6:1364224183.636179:0:1558:0:(quota_interface.c:482:quota_chk_acq_common()) recheck quota with rc: 0, cycle: 10 00040000:04000000:6:1364224183.636181:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00040000:04000000:6:1364224183.636190:0:1558:0:(quota_interface.c:345:quota_check_common()) count=256 lqs_pending=6340608 qd_count=5087232 isblk=2 mb=0 pending[1]=1056768 00040000:04000000:6:1364224183.636193:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00002000:04000000:6:1364224183.636199:0:1558:0:(fsfilt-ldiskfs.c:1601:fsfilt_ldiskfs_quotactl()) quotactl command 0x800007, id 10311, type 0 failed: -3 00040000:04000000:6:1364224183.636205:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00040000:04000000:6:1364224183.636209:0:1558:0:(quota_context.c:347:check_cur_qunit()) type: b, limit: 50417631232, usage: 50416787456, pending_write: 6340608, record: 2097152, qunit_sz: 1048576, tune_sz: 524288, ret: 0. 00040000:04000000:6:1364224183.636214:0:1558:0:(quota_interface.c:482:quota_chk_acq_common()) recheck quota with rc: 0, cycle: 11 00040000:04000000:6:1364224183.636217:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00040000:04000000:6:1364224183.636224:0:1558:0:(quota_interface.c:345:quota_check_common()) count=256 lqs_pending=6340608 qd_count=5087232 isblk=2 mb=0 pending[1]=1056768 00040000:04000000:6:1364224183.636227:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00002000:04000000:6:1364224183.636231:0:1558:0:(fsfilt-ldiskfs.c:1601:fsfilt_ldiskfs_quotactl()) quotactl command 0x800007, id 10311, type 0 failed: -3 00040000:04000000:6:1364224183.636239:0:1558:0:(quota_adjust_qunit.c:172:quota_search_lqs()) lqs(ffff810846b6a280) id(19302) flag(2147483649) type(g) bunit(1048576) btune(524288) iunit(5120) itune(2560) lqs_bwrite_pending(6340608) lqs_iwrite_pending(0) ino_rec(0) blk_rec(2097152)refcount(11): search lqs 00040000:04000000:6:1364224183.636243:0:1558:0:(quota_context.c:347:check_cur_qunit()) type: b, limit: 50417631232, usage: 50416787456, pending_write: 6340608, record: 2097152, qunit_sz: 1048576, tune_sz: 524288, ret: 0. </pre> </div></div> <p>Looks there is pending acquire request (record: 2097152), and local limit will be able to satisfy current write request when the pending acquire done, so OST should wait for the pending reqeust, but it actually didn't wait, so quota_chk_acq_common() recycled many times until the pending request done.</p> <p>I didn't see why OST didn't wait on pending acquire so far, maybe there is something wrong with the lqs_blk_rec calculation? I'll look into it further.</p> <p>Now I think it isn't a problem after I checked the large log for all processes (the small one I checked before is only for single process).</p> <p>When some uid/gid is approaching quota limit, the qunit size will be shrinked to the minimal size (1MB), and each quota acquire can only acquire 1MB limit, if there are many OST threads writing to the same uid/gid at this time, the acquired limit will be consumed very soon, so some writing threads have to recycle several times in the quota_chk_acq_common() until get required quota limit or fail with -EDQUOT at the end.</p> <p>Is it possible that this could cause performance issues when the UID/GID is approaching the quota limit? Do you know if there have ever been performance tests run for this case?</p> <p>Thanks.</p> <p>Yes, when approaching limit, write performance will be lower because quota unit is very small when approaching limit, and there will be much more quota acquire/release operations between ost &amp; mdt, even worse, the write operation has to be blocked on the quota acquire operation sometimes. We didn't measure the performance in such unusual case.</p> <p>Closing old bug.</p> Development Rank 1|hzvbtj: Rank (Obsolete) 5479 Severity