[#LU-4505] invalid "Disk quota exceed" error

[LU-4505] invalid "Disk quota exceed" error Created: 17/Jan/14 Updated: 15/Jul/14 Resolved: 21/Feb/14
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.4.0
Fix Version/s:	Lustre 2.6.0, Lustre 2.5.1

Type:

Bug

Priority:

Major

Reporter:

Mahmoud Hanafi

Assignee:

Niu Yawei (Inactive)

Resolution:

Fixed

Votes:

Labels:

mn4

Issue Links:

Related

Severity:

Rank (Obsolete):

12323

Description

User is getting "Disk quota Exceeded" but has lots of quota available.

Uploaded following file to ftp site.
kferschw.debug.trace.OSS.gz
kferschw.debug.trace.mds

The following was writing to OST00000 stripe dir.
pfe21.kferschw 125> dd if=/dev/zero of=testfile2 bs=1M count=1000
dd: writing `testfile2': Disk quota exceeded

— STRACE OUTPUT----
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = -1 EDQUOT (Disk quota exceeded)

----------------------- pfe21.kferschw Disk quotas for Filesystem kbytes /nobackupp7 386410032 nbp7-MDT0000_UUID
35788 nbp7-OST0000_UUID
3190116 nbp7-OST0001_UUID
5394136 nbp7-OST0002_UUID
4622304 nbp7-OST0003_UUID
4381244 nbp7-OST0004_UUID
5825544 nbp7-OST0005_UUID
4299436 nbp7-OST0006_UUID
3977892 nbp7-OST0007_UUID
4380728 nbp7-OST0008_UUID
4735676 nbp7-OST0009_UUID
4288472 nbp7-OST000a_UUID
3844268 nbp7-OST000b_UUID
4673420 nbp7-OST000c_UUID
4812004 nbp7-OST000d_UUID
4223056 nbp7-OST000e_UUID
5089004 nbp7-OST000f_UUID
4159372 nbp7-OST0010_UUID
5757832 nbp7-OST0011_UUID
4379564 nbp7-OST0012_UUID
3740432 nbp7-OST0013_UUID
4015232 nbp7-OST0014_UUID
4711956 nbp7-OST0015_UUID
4962784 nbp7-OST0016_UUID
4100436 nbp7-OST0017_UUID
4939728 nbp7-OST0018_UUID
4437648 nbp7-OST0019_UUID
4205352 nbp7-OST001a_UUID
3489284 nbp7-OST001b_UUID
3948264 nbp7-OST001c_UUID
4752300 nbp7-OST001d_UUID
3340532 nbp7-OST001e_UUID
3846352 nbp7-OST001f_UUID
3330368 nbp7-OST0020_UUID
4285776 nbp7-OST0021_UUID
4253720 nbp7-OST0022_UUID
3651620 nbp7-OST0023_UUID
4683028 nbp7-OST0024_UUID
4448032 nbp7-OST0025_UUID
3814340 nbp7-OST0026_UUID
4226044 nbp7-OST0027_UUID
5047608 nbp7-OST0028_UUID
4488580 nbp7-OST0029_UUID
4774184 nbp7-OST002a_UUID
3558388 nbp7-OST002b_UUID
29970680 nbp7-OST002c_UUID
3959396 nbp7-OST002d_UUID
4656420 nbp7-OST002e_UUID
4902936 nbp7-OST002f_UUID
2409460 nbp7-OST0030_UUID
4456620 nbp7-OST0031_UUID
4433440 nbp7-OST0032_UUID
5149052 nbp7-OST0033_UUID
4795664 nbp7-OST0034_UUID
4181592 nbp7-OST0035_UUID
5521136* nbp7-OST0036_UUID
4305644 nbp7-OST0037_UUID
3028780 nbp7-OST0038_UUID
4113972 nbp7-OST0039_UUID
3815152 nbp7-OST003a_UUID
4591096 nbp7-OST003b_UUID
3615712 nbp7-OST003c_UUID
4334768 nbp7-OST003d_UUID
4045872 nbp7-OST003e_UUID
4681704 nbp7-OST003f_UUID
4063240 nbp7-OST0040_UUID
5356572 nbp7-OST0041_UUID
4023020 nbp7-OST0042_UUID
3337968 nbp7-OST0043_UUID
2996308 nbp7-OST0044_UUID
4040072 nbp7-OST0045_UUID
6232336 nbp7-OST0046_UUID
3855372 nbp7-OST0047_UUID
3867100 nbp7-OST0048_UUID
3972944 nbp7-OST0049_UUID
4530920 nbp7-OST004a_UUID
3761008 nbp7-OST004b_UUID
3266156 nbp7-OST004c_UUID
5210620 nbp7-OST004d_UUID
4917344 nbp7-OST004e_UUID
4304916 nbp7-OST004f_UUID
3327960 nbp7-OST0050_UUID
4778340 nbp7-OST0051_UUID
5910632 nbp7-OST0052_UUID
4212080 nbp7-OST0053_UUID
2354184 -----------------------------------
124> lfs quota -v -u kferschw /nobackupp7
user kferschw (uid 12262):
quota limit grace files quota limit grace
530000000 1100000000 - 66753 75000 150000 -
- 0 - 66753 - 67496 -
- 3190124 - - - - -
- 5653208 - - - - -
- 4623328 - - - - -
- 4382336 - - - - -
- 5827592 - - - - -
- 4559532 - - - - -
- 4236964 - - - - -
- 4639800 - - - - -
- 4735944 - - - - -
- 4290520 - - - - -
- 3846316 - - - - -
- 5718924 - - - - -
- 5759204 - - - - -
- 5138516 - - - - -
- 5090340 - - - - -
- 4160396 - - - - -
- 6291456 - - - - -
- 4638636 - - - - -
- 3742480 - - - - -
- 4015444 - - - - -
- 4712980 - - - - -
- 4964832 - - - - -
- 4360532 - - - - -
- 5920724 - - - - -
- 4437968 - - - - -
- 4206656 - - - - -
- 3491208 - - - - -
- 3949288 - - - - -
- 5012396 - - - - -
- 4321524 - - - - -
- 3847376 - - - - -
- 3332416 - - - - -
- 4286800 - - - - -
- 4254744 - - - - -
- 3910692 - - - - -
- 4943124 - - - - -
- 4449056 - - - - -
- 3815364 - - - - -
- 4227596 - - - - -
- 5307704 - - - - -
- 4489604 - - - - -
- 4775208 - - - - -
- 3560436 - - - - -
- 30950652 - - - - -
- 3959444 - - - - -
- 4656488 - - - - -
- 4903960 - - - - -
- 3390452 - - - - -
- 4457704 - - - - -
- 4434684 - - - - -
- 5150076 - - - - -
- 5760276 - - - - -
- 4181772 - - - - -
- 5521136 - - - - -
- 4307692 - - - - -
- 4008752 - - - - -
- 4116020 - - - - -
- 3816176 - - - - -
- 4592120 - - - - -
- 3874784 - - - - -
- 4335792 - - - - -
- 4046780 - - - - -
- 4682996 - - - - -
- 4281356 - - - - -
- 5358620 - - - - -
- 4024044 - - - - -
- 3340016 - - - - -
- 3977304 - - - - -
- 4041112 - - - - -
- 6234384 - - - - -
- 3856396 - - - - -
- 4127196 - - - - -
- 3974192 - - - - -
- 4532968 - - - - -
- 3762032 - - - - -
- 4247152 - - - - -
- 5211644 - - - - -
- 4919392 - - - - -
- 4305948 - - - - -
- 4311012 - - - - -
- 4778672 - - - - -
- 5911580 - - - - -
- 4213104 - - - - -
- 2580492 - - - - -

Comments

Comment by Peter Jones [ 17/Jan/14 ]

Niu

Could you please look into this one?

Thanks

Peter

Comment by Niu Yawei (Inactive) [ 20/Jan/14 ]

What kind of operations did you do before this happen? Was the limit for the kferschw bumped to current value (1100000000) from a smaller one? Thanks.

Comment by Mahmoud Hanafi [ 20/Jan/14 ]

The user report this issue. The inability to write has been inconsistent. The user sometime was able to write 25GB file to the filesystem. When the files was deleted he couldn't write a much smaller file.

After I took the debug logs I set the user quota to Zero and then set it back. I ran into the same quota issue. Although I didn't change the (1100000000).

There are other users who have the same issue with quota on this filesystem. This is the only 2.4.x filesystem we have so far.

Comment by Niu Yawei (Inactive) [ 21/Jan/14 ]

hi, Mahmoud

Which means problem can be reproduced even when you reset the quota limit? Could you collect debug log on MDT and the OST with D_TRACE & D_QUOTA enabled when you reset limit (set limit to zero then set back)? Thanks.

Comment by Mahmoud Hanafi [ 21/Jan/14 ]

I will see I can reproduce it. Did look at the logs I uploaded

Comment by Mahmoud Hanafi [ 21/Jan/14 ]

i have uploaded the following debug trace files to the ftp site.
debug.lustre.service182.oss.gz
debug.lustre.service180.mds.gz
debug.lustre1.service182.oss.gz
debug.lustre1.service180.mds.gz

Comment by Niu Yawei (Inactive) [ 22/Jan/14 ]

Thank you, Mahmoud. The log shows that only the OST0000 has such problem, I suspect it because the edquot flag on OST0000 was set mistakenly by some sort of race.

Comment by Niu Yawei (Inactive) [ 22/Jan/14 ]

http://review.whamcloud.com/8954

Comment by Mahmoud Hanafi [ 22/Jan/14 ]

I had create a directory that was only 1 striped fixed on ost00000 to make easier to debug. I think that is why you only saw ost00000.

Comment by Jay Lan (Inactive) [ 22/Jan/14 ]

Hi Niu,

Is the extra check you removed in 8954 not needed at the first place? Could removing the check open door to a different race condition?

Comment by Niu Yawei (Inactive) [ 23/Jan/14 ]

Hi, Jay, I can't think of any other race for now, let's see how patch inspectors think.

Comment by Bob Glossman (Inactive) [ 19/Feb/14 ]

backport to b2_5:
http://review.whamcloud.com/9315

Comment by Peter Jones [ 21/Feb/14 ]

Landed for 2.5.1 and 2.6

Comment by Niu Yawei (Inactive) [ 15/Jul/14 ]

b2_4: http://review.whamcloud.com/11100

Generated at Sat Feb 10 01:43:19 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.

[LU-4505] invalid "Disk quota exceed" error Created: 17/Jan/14 Updated: 15/Jul/14 Resolved: 21/Feb/14