Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-4505] invalid "Disk quota exceed" error https://jira.whamcloud.com/browse/LU-4505 Lustre <p>User is getting "Disk quota Exceeded" but has lots of quota available. </p> <p>Uploaded following file to ftp site.<br/> kferschw.debug.trace.OSS.gz<br/> kferschw.debug.trace.mds</p> <p>The following was writing to OST00000 stripe dir. <br/> pfe21.kferschw 125&gt; dd if=/dev/zero of=testfile2 bs=1M count=1000<br/> dd: writing `testfile2': Disk quota exceeded</p> <p>&#8212; STRACE OUTPUT----<br/> write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = -1 EDQUOT (Disk quota exceeded)</p> <p>----------------------------------------------------------<br/> pfe21.kferschw 124&gt; lfs quota -v -u kferschw /nobackupp7<br/> Disk quotas for user kferschw (uid 12262):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /nobackupp7 386410032 530000000 1100000000 - 66753 75000 150000 -<br/> nbp7-MDT0000_UUID<br/> 35788 - 0 - 66753 - 67496 -<br/> nbp7-OST0000_UUID<br/> 3190116 - 3190124 - - - - -<br/> nbp7-OST0001_UUID<br/> 5394136 - 5653208 - - - - -<br/> nbp7-OST0002_UUID<br/> 4622304 - 4623328 - - - - -<br/> nbp7-OST0003_UUID<br/> 4381244 - 4382336 - - - - -<br/> nbp7-OST0004_UUID<br/> 5825544 - 5827592 - - - - -<br/> nbp7-OST0005_UUID<br/> 4299436 - 4559532 - - - - -<br/> nbp7-OST0006_UUID<br/> 3977892 - 4236964 - - - - -<br/> nbp7-OST0007_UUID<br/> 4380728 - 4639800 - - - - -<br/> nbp7-OST0008_UUID<br/> 4735676 - 4735944 - - - - -<br/> nbp7-OST0009_UUID<br/> 4288472 - 4290520 - - - - -<br/> nbp7-OST000a_UUID<br/> 3844268 - 3846316 - - - - -<br/> nbp7-OST000b_UUID<br/> 4673420 - 5718924 - - - - -<br/> nbp7-OST000c_UUID<br/> 4812004 - 5759204 - - - - -<br/> nbp7-OST000d_UUID<br/> 4223056 - 5138516 - - - - -<br/> nbp7-OST000e_UUID<br/> 5089004 - 5090340 - - - - -<br/> nbp7-OST000f_UUID<br/> 4159372 - 4160396 - - - - -<br/> nbp7-OST0010_UUID<br/> 5757832 - 6291456 - - - - -<br/> nbp7-OST0011_UUID<br/> 4379564 - 4638636 - - - - -<br/> nbp7-OST0012_UUID<br/> 3740432 - 3742480 - - - - -<br/> nbp7-OST0013_UUID<br/> 4015232 - 4015444 - - - - -<br/> nbp7-OST0014_UUID<br/> 4711956 - 4712980 - - - - -<br/> nbp7-OST0015_UUID<br/> 4962784 - 4964832 - - - - -<br/> nbp7-OST0016_UUID<br/> 4100436 - 4360532 - - - - -<br/> nbp7-OST0017_UUID<br/> 4939728 - 5920724 - - - - -<br/> nbp7-OST0018_UUID<br/> 4437648 - 4437968 - - - - -<br/> nbp7-OST0019_UUID<br/> 4205352 - 4206656 - - - - -<br/> nbp7-OST001a_UUID<br/> 3489284 - 3491208 - - - - -<br/> nbp7-OST001b_UUID<br/> 3948264 - 3949288 - - - - -<br/> nbp7-OST001c_UUID<br/> 4752300 - 5012396 - - - - -<br/> nbp7-OST001d_UUID<br/> 3340532 - 4321524 - - - - -<br/> nbp7-OST001e_UUID<br/> 3846352 - 3847376 - - - - -<br/> nbp7-OST001f_UUID<br/> 3330368 - 3332416 - - - - -<br/> nbp7-OST0020_UUID<br/> 4285776 - 4286800 - - - - -<br/> nbp7-OST0021_UUID<br/> 4253720 - 4254744 - - - - -<br/> nbp7-OST0022_UUID<br/> 3651620 - 3910692 - - - - -<br/> nbp7-OST0023_UUID<br/> 4683028 - 4943124 - - - - -<br/> nbp7-OST0024_UUID<br/> 4448032 - 4449056 - - - - -<br/> nbp7-OST0025_UUID<br/> 3814340 - 3815364 - - - - -<br/> nbp7-OST0026_UUID<br/> 4226044 - 4227596 - - - - -<br/> nbp7-OST0027_UUID<br/> 5047608 - 5307704 - - - - -<br/> nbp7-OST0028_UUID<br/> 4488580 - 4489604 - - - - -<br/> nbp7-OST0029_UUID<br/> 4774184 - 4775208 - - - - -<br/> nbp7-OST002a_UUID<br/> 3558388 - 3560436 - - - - -<br/> nbp7-OST002b_UUID<br/> 29970680 - 30950652 - - - - -<br/> nbp7-OST002c_UUID<br/> 3959396 - 3959444 - - - - -<br/> nbp7-OST002d_UUID<br/> 4656420 - 4656488 - - - - -<br/> nbp7-OST002e_UUID<br/> 4902936 - 4903960 - - - - -<br/> nbp7-OST002f_UUID<br/> 2409460 - 3390452 - - - - -<br/> nbp7-OST0030_UUID<br/> 4456620 - 4457704 - - - - -<br/> nbp7-OST0031_UUID<br/> 4433440 - 4434684 - - - - -<br/> nbp7-OST0032_UUID<br/> 5149052 - 5150076 - - - - -<br/> nbp7-OST0033_UUID<br/> 4795664 - 5760276 - - - - -<br/> nbp7-OST0034_UUID<br/> 4181592 - 4181772 - - - - -<br/> nbp7-OST0035_UUID<br/> 5521136* - 5521136 - - - - -<br/> nbp7-OST0036_UUID<br/> 4305644 - 4307692 - - - - -<br/> nbp7-OST0037_UUID<br/> 3028780 - 4008752 - - - - -<br/> nbp7-OST0038_UUID<br/> 4113972 - 4116020 - - - - -<br/> nbp7-OST0039_UUID<br/> 3815152 - 3816176 - - - - -<br/> nbp7-OST003a_UUID<br/> 4591096 - 4592120 - - - - -<br/> nbp7-OST003b_UUID<br/> 3615712 - 3874784 - - - - -<br/> nbp7-OST003c_UUID<br/> 4334768 - 4335792 - - - - -<br/> nbp7-OST003d_UUID<br/> 4045872 - 4046780 - - - - -<br/> nbp7-OST003e_UUID<br/> 4681704 - 4682996 - - - - -<br/> nbp7-OST003f_UUID<br/> 4063240 - 4281356 - - - - -<br/> nbp7-OST0040_UUID<br/> 5356572 - 5358620 - - - - -<br/> nbp7-OST0041_UUID<br/> 4023020 - 4024044 - - - - -<br/> nbp7-OST0042_UUID<br/> 3337968 - 3340016 - - - - -<br/> nbp7-OST0043_UUID<br/> 2996308 - 3977304 - - - - -<br/> nbp7-OST0044_UUID<br/> 4040072 - 4041112 - - - - -<br/> nbp7-OST0045_UUID<br/> 6232336 - 6234384 - - - - -<br/> nbp7-OST0046_UUID<br/> 3855372 - 3856396 - - - - -<br/> nbp7-OST0047_UUID<br/> 3867100 - 4127196 - - - - -<br/> nbp7-OST0048_UUID<br/> 3972944 - 3974192 - - - - -<br/> nbp7-OST0049_UUID<br/> 4530920 - 4532968 - - - - -<br/> nbp7-OST004a_UUID<br/> 3761008 - 3762032 - - - - -<br/> nbp7-OST004b_UUID<br/> 3266156 - 4247152 - - - - -<br/> nbp7-OST004c_UUID<br/> 5210620 - 5211644 - - - - -<br/> nbp7-OST004d_UUID<br/> 4917344 - 4919392 - - - - -<br/> nbp7-OST004e_UUID<br/> 4304916 - 4305948 - - - - -<br/> nbp7-OST004f_UUID<br/> 3327960 - 4311012 - - - - -<br/> nbp7-OST0050_UUID<br/> 4778340 - 4778672 - - - - -<br/> nbp7-OST0051_UUID<br/> 5910632 - 5911580 - - - - -<br/> nbp7-OST0052_UUID<br/> 4212080 - 4213104 - - - - -<br/> nbp7-OST0053_UUID<br/> 2354184 - 2580492 - - - - -</p> LU-4505 invalid "Disk quota exceed" error Bug Major Resolved Fixed Niu Yawei Mahmoud Hanafi Fri, 17 Jan 2014 18:08:58 +0000 Tue, 15 Jul 2014 06:39:52 +0000 Fri, 21 Feb 2014 05:30:01 +0000 Lustre 2.4.0 Lustre 2.6.0 Lustre 2.5.1 0 6 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>What kind of operations did you do before this happen? Was the limit for the kferschw bumped to current value (1100000000) from a smaller one? Thanks.</p> <p>The user report this issue. The inability to write has been inconsistent. The user sometime was able to write 25GB file to the filesystem. When the files was deleted he couldn't write a much smaller file.</p> <p>After I took the debug logs I set the user quota to Zero and then set it back. I ran into the same quota issue. Although I didn't change the (1100000000).</p> <p>There are other users who have the same issue with quota on this filesystem. This is the only 2.4.x filesystem we have so far. </p> <p>hi, Mahmoud</p> <p>Which means problem can be reproduced even when you reset the quota limit? Could you collect debug log on MDT and the OST with D_TRACE &amp; D_QUOTA enabled when you reset limit (set limit to zero then set back)? Thanks.</p> <p>I will see I can reproduce it. Did look at the logs I uploaded</p> <p>i have uploaded the following debug trace files to the ftp site.<br/> debug.lustre.service182.oss.gz<br/> debug.lustre.service180.mds.gz<br/> debug.lustre1.service182.oss.gz<br/> debug.lustre1.service180.mds.gz</p> <p>Thank you, Mahmoud. The log shows that only the OST0000 has such problem, I suspect it because the edquot flag on OST0000 was set mistakenly by some sort of race.</p> <p><a href="http://review.whamcloud.com/8954" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8954</a></p> <p>I had create a directory that was only 1 striped fixed on ost00000 to make easier to debug. I think that is why you only saw ost00000.</p> <p>Hi Niu,</p> <p>Is the extra check you removed in 8954 not needed at the first place? Could removing the check open door to a different race condition?</p> <p>Hi, Jay, I can't think of any other race for now, let's see how patch inspectors think.</p> <p>backport to b2_5:<br/> <a href="http://review.whamcloud.com/9315" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9315</a></p> <p>Landed for 2.5.1 and 2.6</p> <p>b2_4: <a href="http://review.whamcloud.com/11100" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11100</a></p> Related Development Rank 1|hzwd2v: Rank (Obsolete) 12323 Severity