mdt0_quota.log
[LU-12557] Wrong MDT inode quota limits lead to quota not enforced Created: 16/Jul/19 Updated: 24/Apr/20 |
|
| Status: | Open |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.10.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical |
| Reporter: | Stephane Thiell | Assignee: | Hongchao Zhang |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Environment: |
CentOS 7 - Lustre 2.10.8 clients and servers |
||
| Attachments: |
mdt0_quota.log
|
| Severity: | 3 |
| Rank (Obsolete): | 9223372036854775807 |
| Description |
|
This morning, we found a group on Oak that was able to get past their inode quota. The limit should be 1.5M inodes, but they were able to get past it at about 1.9M inodes. We have checked that this didn't involve Lustre 2.12 (tested on a 2.10.8 client also just to be sure it was not a 2.10/2.12 compat issue).
[root@oak-gw02 ~]# lfs quota -g oak_hlwill /oak
Disk quotas for grp oak_hlwill (gid 6324):
Filesystem kbytes quota limit grace files quota limit grace
/oak 8533764076 10000000000 10000000000 - 1899647* 1500000 1500000 -
[root@oak-gw02 ~]# lfs quota -v -g oak_hlwill /oak
Disk quotas for grp oak_hlwill (gid 6324):
Filesystem kbytes quota limit grace files quota limit grace
/oak 8533764076 10000000000 10000000000 - 1899647* 1500000 1500000 -
oak-MDT0000_UUID
0 - 0 - 0 - 1227848 -
oak-MDT0001_UUID
191560 - 0 - 1899647 - 1900270 -
oak-OST0000_UUID
17351952 - 31759736 - - - - -
oak-OST0001_UUID
66026588 - 81822144 - - - - -
oak-OST0002_UUID
14922776 - 31511992 - - - - -
oak-OST0003_UUID
15177156 - 31764608 - - - - -
oak-OST0004_UUID
24677808 - 40406296 - - - - -
oak-OST0005_UUID
66847632 - 77731728 - - - - -
oak-OST0006_UUID
41358308 - 47813388 - - - - -
oak-OST0007_UUID
51010844 - 62061088 - - - - -
oak-OST0008_UUID
27771804 - 44114220 - - - - -
oak-OST0009_UUID
14601076 - 19432308 - - - - -
oak-OST000a_UUID
34389472 - 47193616 - - - - -
oak-OST000b_UUID
37729512 - 47398016 - - - - -
oak-OST000c_UUID
19129260 - 31648056 - - - - -
oak-OST000d_UUID
86069244 - 94557420 - - - - -
oak-OST000e_UUID
102349016 - 110564620 - - - - -
oak-OST000f_UUID
54054828 - 63654276 - - - - -
oak-OST0010_UUID
65474316 - 78720760 - - - - -
oak-OST0011_UUID
35672936 - 47747340 - - - - -
oak-OST0012_UUID
34334216 - 48068880 - - - - -
oak-OST0013_UUID
174346708 - 189803564 - - - - -
oak-OST0014_UUID
35482060 - 48985040 - - - - -
oak-OST0015_UUID
30785012 - 47307496 - - - - -
oak-OST0016_UUID
14288204 - 30010196 - - - - -
oak-OST0017_UUID
160723328 - 170963160 - - - - -
oak-OST0018_UUID
29121536 - 45781220 - - - - -
oak-OST0019_UUID
36321604 - 50362328 - - - - -
oak-OST001a_UUID
20412724 - 31637720 - - - - -
oak-OST001b_UUID
182895796 - 196926944 - - - - -
oak-OST001c_UUID
19553908 - 32030544 - - - - -
oak-OST001d_UUID
62042768 - 78075652 - - - - -
oak-OST001e_UUID
44483932 - 61192240 - - - - -
oak-OST001f_UUID
56387556 - 62735272 - - - - -
oak-OST0020_UUID
48325020 - 52707804 - - - - -
oak-OST0021_UUID
21045360 - 32513732 - - - - -
oak-OST0022_UUID
24929796 - 32562208 - - - - -
oak-OST0023_UUID
16102080 - 32269368 - - - - -
oak-OST0024_UUID
33986196 - 48985176 - - - - -
oak-OST0025_UUID
13570580 - 29364152 - - - - -
oak-OST0026_UUID
45386672 - 62043520 - - - - -
oak-OST0027_UUID
13126336 - 29150520 - - - - -
oak-OST0028_UUID
20959648 - 32092356 - - - - -
oak-OST0029_UUID
53692068 - 63209868 - - - - -
oak-OST002a_UUID
67681888 - 77166948 - - - - -
oak-OST002b_UUID
24941128 - 31475784 - - - - -
oak-OST002c_UUID
22913216 - 32134972 - - - - -
oak-OST002d_UUID
15253452 - 31461332 - - - - -
oak-OST002e_UUID
31644152 - 47380988 - - - - -
oak-OST002f_UUID
77376284 - 93805364 - - - - -
oak-OST0030_UUID
33129164 - 46462924 - - - - -
oak-OST0031_UUID
32701444 - 47223908 - - - - -
oak-OST0032_UUID
55045664 - 63461196 - - - - -
oak-OST0033_UUID
19591076 - 30847104 - - - - -
oak-OST0034_UUID
25803956 - 32684668 - - - - -
oak-OST0035_UUID
17417308 - 32159544 - - - - -
oak-OST0036_UUID
34644580 - 42654224 - - - - -
oak-OST0037_UUID
55375144 - 63712260 - - - - -
oak-OST0038_UUID
17038824 - 31374628 - - - - -
oak-OST0039_UUID
39546648 - 47353392 - - - - -
oak-OST003a_UUID
102553208 - 110268964 - - - - -
oak-OST003b_UUID
28840752 - 44928480 - - - - -
oak-OST003c_UUID
49202920 - 63133228 - - - - -
oak-OST003d_UUID
49559824 - 63497608 - - - - -
oak-OST003e_UUID
13722604 - 30024312 - - - - -
oak-OST003f_UUID
70378068 - 78122168 - - - - -
oak-OST0040_UUID
68462244 - 77360832 - - - - -
oak-OST0041_UUID
57653700 - 63235376 - - - - -
oak-OST0042_UUID
20172548 - 32054296 - - - - -
oak-OST0043_UUID
37357332 - 48946684 - - - - -
oak-OST0044_UUID
126518512 - 141769980 - - - - -
oak-OST0045_UUID
67839960 - 78785684 - - - - -
oak-OST0046_UUID
39016316 - 55793244 - - - - -
oak-OST0047_UUID
17682040 - 31681328 - - - - -
oak-OST0048_UUID
240605844 - 251938788 - - - - -
oak-OST0049_UUID
30177248 - 46168020 - - - - -
oak-OST004a_UUID
44678516 - 58587588 - - - - -
oak-OST004b_UUID
61939084 - 78400116 - - - - -
oak-OST004c_UUID
21858052 - 34955536 - - - - -
oak-OST004d_UUID
138522544 - 154789120 - - - - -
oak-OST004e_UUID
49504748 - 63841892 - - - - -
oak-OST004f_UUID
130926420 - 146466620 - - - - -
oak-OST0050_UUID
57729864 - 63365516 - - - - -
oak-OST0051_UUID
48460656 - 63959460 - - - - -
oak-OST0052_UUID
94210388 - 109956288 - - - - -
oak-OST0053_UUID
64437928 - 78429300 - - - - -
oak-OST0054_UUID
42852680 - 53818780 - - - - -
oak-OST0055_UUID
183838744 - 199938748 - - - - -
oak-OST0056_UUID
39198436 - 46140168 - - - - -
oak-OST0057_UUID
68744424 - 84933852 - - - - -
oak-OST0058_UUID
31213976 - 47337560 - - - - -
oak-OST0059_UUID
57913780 - 74479388 - - - - -
oak-OST005a_UUID
82979688 - 97595716 - - - - -
oak-OST005b_UUID
67481288 - 79103856 - - - - -
oak-OST005c_UUID
97845408 - 112712476 - - - - -
oak-OST005d_UUID
71769900 - 88129176 - - - - -
oak-OST005e_UUID
40552544 - 46876768 - - - - -
oak-OST005f_UUID
30870984 - 46878048 - - - - -
oak-OST0060_UUID
54666344 - 0 - - - - -
oak-OST0061_UUID
57431844 - 0 - - - - -
oak-OST0062_UUID
137714524 - 0 - - - - -
oak-OST0063_UUID
22874128 - 0 - - - - -
oak-OST0064_UUID
76953932 - 0 - - - - -
oak-OST0065_UUID
103531832 - 0 - - - - -
oak-OST0066_UUID
71837956 - 0 - - - - -
oak-OST0067_UUID
46396392 - 0 - - - - -
oak-OST0068_UUID
132340672 - 0 - - - - -
oak-OST0069_UUID
39410204 - 0 - - - - -
oak-OST006a_UUID
43926100 - 0 - - - - -
oak-OST006b_UUID
154165452 - 0 - - - - -
oak-OST006c_UUID
210254328 - 0 - - - - -
oak-OST006d_UUID
35547240 - 0 - - - - -
oak-OST006e_UUID
49122000 - 0 - - - - -
oak-OST006f_UUID
44907184 - 0 - - - - -
oak-OST0070_UUID
43110756 - 0 - - - - -
oak-OST0071_UUID
50232656 - 0 - - - - -
oak-OST0072_UUID
244608820 - 0 - - - - -
oak-OST0073_UUID
90048308 - 0 - - - - -
oak-OST0074_UUID
78561112 - 0 - - - - -
oak-OST0075_UUID
81974144 - 0 - - - - -
oak-OST0076_UUID
235628012 - 0 - - - - -
oak-OST0077_UUID
279483236 - 0 - - - - -
oak-OST0078_UUID
139394324 - 0 - - - - -
oak-OST0079_UUID
122266916 - 0 - - - - -
oak-OST007a_UUID
194638420 - 0 - - - - -
oak-OST007b_UUID
131759012 - 0 - - - - -
oak-OST007c_UUID
342335656 - 0 - - - - -
oak-OST007d_UUID
130494584 - 0 - - - - -
oak-OST007e_UUID
1632 - 0 - - - - -
oak-OST007f_UUID
125044 - 0 - - - - -
oak-OST0080_UUID
52284 - 0 - - - - -
oak-OST0081_UUID
70460 - 0 - - - - -
oak-OST0082_UUID
15648 - 0 - - - - -
oak-OST0083_UUID
510460 - 0 - - - - -
oak-OST0084_UUID
138332 - 0 - - - - -
oak-OST0085_UUID
84700 - 0 - - - - -
oak-OST0086_UUID
1024384 - 0 - - - - -
oak-OST0087_UUID
76132 - 0 - - - - -
oak-OST0088_UUID
26312 - 0 - - - - -
oak-OST0089_UUID
68328 - 0 - - - - -
oak-OST008a_UUID
59324 - 0 - - - - -
oak-OST008b_UUID
268480 - 0 - - - - -
oak-OST008c_UUID
41144 - 0 - - - - -
oak-OST008d_UUID
267472 - 0 - - - - -
oak-OST008e_UUID
120212 - 0 - - - - -
oak-OST008f_UUID
711344 - 0 - - - - -
Total allocated inode limit: 3128118, total allocated block limit: 6298444772
# clush -bw@mds 'lctl get_param osd-*.*.quota_slave.info' --------------- oak-md1-s1 --------------- osd-ldiskfs.oak-MDT0001.quota_slave.info= target name: oak-MDT0001 pool ID: 0 type: md quota enabled: g conn to master: setup space acct: ug user uptodate: glb[0],slv[0],reint[0] group uptodate: glb[1],slv[1],reint[0] project uptodate: glb[0],slv[0],reint[0] --------------- oak-md1-s2 --------------- osd-ldiskfs.oak-MDT0000.quota_slave.info= target name: oak-MDT0000 pool ID: 0 type: md quota enabled: g conn to master: setup space acct: ug user uptodate: glb[0],slv[0],reint[0] group uptodate: glb[1],slv[1],reint[0] project uptodate: glb[0],slv[0],reint[0] Group quota is not enforced at this time, I was able to create new files in this group. Any suggestion? |
| Comments |
| Comment by Peter Jones [ 16/Jul/19 ] |
|
Hongchao Can you please advise? Thanks Peter |
| Comment by Hongchao Zhang [ 18/Jul/19 ] |
|
Hi Stephane, Could you please run the following command and collect the logs? Thanks! # clear previous debug log lctl dk >/path/to/log lctl set_param debug=quota #enable the group quota on MDT0 and MDT1 again. #run at MGS lctl conf_param lustre.quota.mdt=g #create some files to produce the logs related to the quota # collect the debug log lctl dk >/path/to/log |
| Comment by Stephane Thiell [ 18/Jul/19 ] |
|
Hi Hongchao, Thanks! Sorry I didn't think about that at first... because this morning, the group had purged most of their files. They likely hit the quota limit at some point. Now I tried to reproduce myself by creating a lot of files in this group but I hit the 1500000 inode quota limit right away. [root@oak-gw02 ~]# lfs quota -v -g oak_hlwill /oak
Disk quotas for grp oak_hlwill (gid 6324):
Filesystem kbytes quota limit grace files quota limit grace
/oak 4881747104 10000000000 10000000000 - 1500000* 1500000 1500000 -
oak-MDT0000_UUID
0 - 0 - 0 - 1227848 -
oak-MDT0001_UUID
128092 - 0 - 1500000* - 1500000 -
I'll double-check with this group to see what they did to create their 1.9M previous files, and get the some debug logs with quota if we can reproduce. |
| Comment by Stephane Thiell [ 19/Jul/19 ] |
|
Hey Hongchao, I've seen the issue again, but this time on the disk quota (not inode quota). [root@oak-gw02 groups]# lfs quota -g -v oak_euan /oak
Disk quotas for grp oak_euan (gid 3593):
Filesystem kbytes quota limit grace files quota limit grace
/oak 90529424132* 90000000000 90000000000 - 9143793 13500000 13500000 -
oak-MDT0000_UUID
2103700 - 0 - 9143793 - 9934775 -
oak-MDT0001_UUID
0 - 0 - 0 - 191359 -
oak-OST0000_UUID
922165692 - 985540596 - - - - -
oak-OST0001_UUID
941759692 - 1007894448 - - - - -
oak-OST0002_UUID
1325724492 - 1371668452 - - - - -
oak-OST0003_UUID
1446155268 - 1512541756 - - - - -
oak-OST0004_UUID
933501476 - 999966576 - - - - -
oak-OST0005_UUID
2121024772 - 2140332116 - - - - -
oak-OST0006_UUID
830852836 - 854082304 - - - - -
oak-OST0007_UUID
1416922556 - 1483828820 - - - - -
oak-OST0008_UUID
757715872 - 792785956 - - - - -
oak-OST0009_UUID
833452856 - 861581188 - - - - -
oak-OST000a_UUID
1452150068 - 1517440656 - - - - -
oak-OST000b_UUID
1029292520 - 1058188204 - - - - -
oak-OST000c_UUID
2387483376 - 2432940376 - - - - -
oak-OST000d_UUID
1007138740 - 1031523196 - - - - -
oak-OST000e_UUID
1594170044 - 1660525520 - - - - -
oak-OST000f_UUID
1192713180 - 1210611472 - - - - -
oak-OST0010_UUID
1003881580 - 1041417124 - - - - -
oak-OST0011_UUID
1403741356 - 1470375812 - - - - -
oak-OST0012_UUID
1018774136 - 1072963072 - - - - -
oak-OST0013_UUID
515825980 - 581942564 - - - - -
oak-OST0014_UUID
456958968 - 523207072 - - - - -
oak-OST0015_UUID
522295504 - 588449664 - - - - -
oak-OST0016_UUID
785748008 - 841273980 - - - - -
oak-OST0017_UUID
2033941788 - 2065476768 - - - - -
oak-OST0018_UUID
547414196 - 589657608 - - - - -
oak-OST0019_UUID
1290115892 - 1315785116 - - - - -
oak-OST001a_UUID
1007797040 - 1074836896 - - - - -
oak-OST001b_UUID
1269122388 - 1335816068 - - - - -
oak-OST001c_UUID
1094766304 - 1161014680 - - - - -
oak-OST001d_UUID
661283484 - 678420644 - - - - -
oak-OST001e_UUID
624059332 - 660133496 - - - - -
oak-OST001f_UUID
472329160 - 525802612 - - - - -
oak-OST0020_UUID
939712204 - 950039856 - - - - -
oak-OST0021_UUID
997890408 - 1064897992 - - - - -
oak-OST0022_UUID
1974634948 - 2040889860 - - - - -
oak-OST0023_UUID
932830612 - 999534176 - - - - -
oak-OST0024_UUID
2591236296 - 2627343104 - - - - -
oak-OST0025_UUID
1379039480 - 1397272964 - - - - -
oak-OST0026_UUID
744016844 - 775453992 - - - - -
oak-OST0027_UUID
669919288 - 732961268 - - - - -
oak-OST0028_UUID
1329384316 - 1383304444 - - - - -
oak-OST0029_UUID
752256752 - 819127776 - - - - -
oak-OST002a_UUID
621938388 - 638002524 - - - - -
oak-OST002b_UUID
701244032 - 740740280 - - - - -
oak-OST002c_UUID
706380496 - 753512980 - - - - -
oak-OST002d_UUID
638167768 - 698833640 - - - - -
oak-OST002e_UUID
422855052 - 484270540 - - - - -
oak-OST002f_UUID
563012176 - 629717128 - - - - -
oak-OST0030_UUID
442252152 - 499957356 - - - - -
oak-OST0031_UUID
618957336 - 685182508 - - - - -
oak-OST0032_UUID
594538388 - 632444460 - - - - -
oak-OST0033_UUID
364698116 - 429575788 - - - - -
oak-OST0034_UUID
316256568 - 368277268 - - - - -
oak-OST0035_UUID
649943944 - 674543316 - - - - -
oak-OST0036_UUID
417007304 - 474549696 - - - - -
oak-OST0037_UUID
685105228 - 751556616 - - - - -
oak-OST0038_UUID
454088760 - 494079616 - - - - -
oak-OST0039_UUID
504511856 - 530288444 - - - - -
oak-OST003a_UUID
423460480 - 438217148 - - - - -
oak-OST003b_UUID
801870700 - 863487908 - - - - -
oak-OST003c_UUID
553644448 - 619898960 - - - - -
oak-OST003d_UUID
1543853000 - 1556897852 - - - - -
oak-OST003e_UUID
383811776 - 444001504 - - - - -
oak-OST003f_UUID
421438624 - 469044764 - - - - -
oak-OST0040_UUID
554667372 - 608879536 - - - - -
oak-OST0041_UUID
568768260 - 614176260 - - - - -
oak-OST0042_UUID
639978980 - 676777052 - - - - -
oak-OST0043_UUID
365046548 - 431664776 - - - - -
oak-OST0044_UUID
390849908 - 442081796 - - - - -
oak-OST0045_UUID
963716568 - 1027625612 - - - - -
oak-OST0046_UUID
601245336 - 643389652 - - - - -
oak-OST0047_UUID
401730192 - 461674284 - - - - -
oak-OST0048_UUID
528791148 - 551094660 - - - - -
oak-OST0049_UUID
909714900 - 949218212 - - - - -
oak-OST004a_UUID
320610096 - 350215304 - - - - -
oak-OST004b_UUID
465336912 - 478234132 - - - - -
oak-OST004c_UUID
481763152 - 548810292 - - - - -
oak-OST004d_UUID
999019300 - 1053994992 - - - - -
oak-OST004e_UUID
574700184 - 641219528 - - - - -
oak-OST004f_UUID
499115136 - 516732680 - - - - -
oak-OST0050_UUID
481925764 - 548883792 - - - - -
oak-OST0051_UUID
668490020 - 735167992 - - - - -
oak-OST0052_UUID
887667604 - 918951156 - - - - -
oak-OST0053_UUID
511044232 - 533751768 - - - - -
oak-OST0054_UUID
641322280 - 707680444 - - - - -
oak-OST0055_UUID
722810608 - 760901740 - - - - -
oak-OST0056_UUID
780337536 - 847174088 - - - - -
oak-OST0057_UUID
506869632 - 573473292 - - - - -
oak-OST0058_UUID
394409164 - 461472076 - - - - -
oak-OST0059_UUID
991874608 - 1036018736 - - - - -
oak-OST005a_UUID
351680568 - 380866784 - - - - -
oak-OST005b_UUID
394870500 - 461979364 - - - - -
oak-OST005c_UUID
333415668 - 364994364 - - - - -
oak-OST005d_UUID
619134820 - 672831612 - - - - -
oak-OST005e_UUID
226679340 - 293637104 - - - - -
oak-OST005f_UUID
299360300 - 354928924 - - - - -
oak-OST0060_UUID
501450052 - 0 - - - - -
oak-OST0061_UUID
245723092 - 0 - - - - -
oak-OST0062_UUID
237817052 - 0 - - - - -
oak-OST0063_UUID
497988600 - 0 - - - - -
oak-OST0064_UUID
271008772 - 0 - - - - -
oak-OST0065_UUID
750523332 - 0 - - - - -
oak-OST0066_UUID
356156588 - 0 - - - - -
oak-OST0067_UUID
431379940 - 0 - - - - -
oak-OST0068_UUID
423980088 - 0 - - - - -
oak-OST0069_UUID
437102468 - 0 - - - - -
oak-OST006a_UUID
354793548 - 0 - - - - -
oak-OST006b_UUID
527046056 - 0 - - - - -
oak-OST006c_UUID
264304292 - 0 - - - - -
oak-OST006d_UUID
561538052 - 0 - - - - -
oak-OST006e_UUID
413233020 - 0 - - - - -
oak-OST006f_UUID
397662008 - 0 - - - - -
oak-OST0070_UUID
362951804 - 0 - - - - -
oak-OST0071_UUID
705492672 - 0 - - - - -
oak-OST0072_UUID
167512940 - 0 - - - - -
oak-OST0073_UUID
176038468 - 0 - - - - -
oak-OST0074_UUID
338074088 - 0 - - - - -
oak-OST0075_UUID
199566968 - 0 - - - - -
oak-OST0076_UUID
721559356 - 0 - - - - -
oak-OST0077_UUID
323730764 - 0 - - - - -
oak-OST0078_UUID
233504028 - 0 - - - - -
oak-OST0079_UUID
292555952 - 0 - - - - -
oak-OST007a_UUID
322945960 - 0 - - - - -
oak-OST007b_UUID
349553564 - 0 - - - - -
oak-OST007c_UUID
166562776 - 0 - - - - -
oak-OST007d_UUID
194382772 - 0 - - - - -
oak-OST007e_UUID
25830772 - 0 - - - - -
oak-OST007f_UUID
31056268 - 0 - - - - -
oak-OST0080_UUID
62835084 - 0 - - - - -
oak-OST0081_UUID
21891996 - 0 - - - - -
oak-OST0082_UUID
43497376 - 0 - - - - -
oak-OST0083_UUID
29844536 - 0 - - - - -
oak-OST0084_UUID
81480528 - 0 - - - - -
oak-OST0085_UUID
78325840 - 0 - - - - -
oak-OST0086_UUID
18636592 - 0 - - - - -
oak-OST0087_UUID
14791544 - 0 - - - - -
oak-OST0088_UUID
13172700 - 0 - - - - -
oak-OST0089_UUID
17864260 - 0 - - - - -
oak-OST008a_UUID
400616124 - 0 - - - - -
oak-OST008b_UUID
130462804 - 0 - - - - -
oak-OST008c_UUID
136703000 - 0 - - - - -
oak-OST008d_UUID
23737664 - 0 - - - - -
oak-OST008e_UUID
25915892 - 0 - - - - -
oak-OST008f_UUID
27335448 - 0 - - - - -
Total allocated inode limit: 10126134, total allocated block limit: 82764432544
I was able to take some debug from MDT0: [root@oak-md1-s2 ~]# lctl dk >/tmp/mdt0 [root@oak-md1-s2 ~]# lctl set_param debug=quota debug=quota [root@oak-md1-s1 ~]# lctl conf_param oak.quota.mdt=g [root@oak-md1-s2 ~]# lctl dk >/tmp/mdt0_quota.log File attached as mdt0_quota.log |
| Comment by Hongchao Zhang [ 20/Jul/19 ] |
|
Hi Stephane, for Lustre 2.10, the disk quota is only enforced on OSTs. Total allocated inode limit: 10126134, total allocated block limit: 82764432544 it shows the same in QMT logs 00040000:04000000:18.0:1563577265.802049:0:17580:0:(qmt_handler.c:65:qmt_get()) $$$ fetch settings qmt:oak-QMT0000 pool:0-md id:3593 enforced:1 hard:13500000 soft:13500000 granted:9934775 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0 00040000:04000000:18.0:1563577265.802053:0:17580:0:(qmt_handler.c:65:qmt_get()) $$$ fetch settings qmt:oak-QMT0000 pool:0-dt id:3593 enforced:1 hard:90000000000 soft:90000000000 granted:82698698300 time:0 qunit:67108864 edquot:0 may_rel:0 revoke:0 from oak-OST0060 to oak-OST008F, there is only some filed occupied by group "oak_euan", but doesn't consume quota. |
| Comment by Stephane Thiell [ 20/Jul/19 ] |
|
Thanks for checking Hongchao, much appreciated! These OSTs were added to the filesystem live at some point. Yesterday, when gathering these logs, I ran lctl conf_param oak.quota.ost=g. Maybe it is required to rerun this command after adding new OSTs? This morning, they are consuming quota: [root@oak-gw02 ~]# lfs quota -v -g oak_euan /oak
Disk quotas for grp oak_euan (gid 3593):
Filesystem kbytes quota limit grace files quota limit grace
/oak 89596213520 90000000000 90000000000 - 9143792 13500000 13500000 -
oak-MDT0000_UUID
2103700 - 0 - 9143792 - 9934775 -
oak-MDT0001_UUID
0 - 0 - 0 - 191359 -
oak-OST0000_UUID
830722372 - 831770948 - - - - -
oak-OST0001_UUID
941759692 - 942808268 - - - - -
oak-OST0002_UUID
1325724492 - 1326773068 - - - - -
oak-OST0003_UUID
1446155268 - 1447203844 - - - - -
oak-OST0004_UUID
933501476 - 934550052 - - - - -
oak-OST0005_UUID
2121024772 - 2122073348 - - - - -
oak-OST0006_UUID
830852836 - 831901412 - - - - -
oak-OST0007_UUID
1416922556 - 1417971132 - - - - -
oak-OST0008_UUID
757715872 - 758764448 - - - - -
oak-OST0009_UUID
833452856 - 834501432 - - - - -
oak-OST000a_UUID
1452150068 - 1453198644 - - - - -
oak-OST000b_UUID
1029292520 - 1030341096 - - - - -
oak-OST000c_UUID
2387483376 - 2388531952 - - - - -
oak-OST000d_UUID
1007138740 - 1008187316 - - - - -
oak-OST000e_UUID
1527078040 - 1528126616 - - - - -
oak-OST000f_UUID
1192713180 - 1193761756 - - - - -
oak-OST0010_UUID
1003881580 - 1004930156 - - - - -
oak-OST0011_UUID
1403741356 - 1404789932 - - - - -
oak-OST0012_UUID
1018774136 - 1019822712 - - - - -
oak-OST0013_UUID
515825980 - 516874556 - - - - -
oak-OST0014_UUID
456958968 - 458007544 - - - - -
oak-OST0015_UUID
522295504 - 523344080 - - - - -
oak-OST0016_UUID
785748008 - 786796584 - - - - -
oak-OST0017_UUID
2033941788 - 2034990364 - - - - -
oak-OST0018_UUID
547414196 - 548462772 - - - - -
oak-OST0019_UUID
1290115892 - 1291164468 - - - - -
oak-OST001a_UUID
1007797040 - 1008845616 - - - - -
oak-OST001b_UUID
1269122388 - 1270170964 - - - - -
oak-OST001c_UUID
1094766304 - 1095814880 - - - - -
oak-OST001d_UUID
661283484 - 662332060 - - - - -
oak-OST001e_UUID
624050504 - 624845764 - - - - -
oak-OST001f_UUID
472329160 - 473377736 - - - - -
oak-OST0020_UUID
939712204 - 940760780 - - - - -
oak-OST0021_UUID
997892292 - 998938984 - - - - -
oak-OST0022_UUID
1974626144 - 1975421380 - - - - -
oak-OST0023_UUID
837539484 - 838588060 - - - - -
oak-OST0024_UUID
2591236296 - 2592284872 - - - - -
oak-OST0025_UUID
1379039480 - 1380088056 - - - - -
oak-OST0026_UUID
744016844 - 745065420 - - - - -
oak-OST0027_UUID
669919288 - 670967864 - - - - -
oak-OST0028_UUID
1097672160 - 1098720736 - - - - -
oak-OST0029_UUID
752256752 - 753305328 - - - - -
oak-OST002a_UUID
621938388 - 622986964 - - - - -
oak-OST002b_UUID
701244032 - 702292608 - - - - -
oak-OST002c_UUID
706380496 - 707429072 - - - - -
oak-OST002d_UUID
638167768 - 639216344 - - - - -
oak-OST002e_UUID
422855052 - 423903628 - - - - -
oak-OST002f_UUID
563012176 - 564060752 - - - - -
oak-OST0030_UUID
442257008 - 443300728 - - - - -
oak-OST0031_UUID
618957336 - 620005912 - - - - -
oak-OST0032_UUID
594538388 - 595586964 - - - - -
oak-OST0033_UUID
386170512 - 387011104 - - - - -
oak-OST0034_UUID
316256568 - 317305144 - - - - -
oak-OST0035_UUID
649943944 - 650992520 - - - - -
oak-OST0036_UUID
417007304 - 418055880 - - - - -
oak-OST0037_UUID
743387420 - 743696108 - - - - -
oak-OST0038_UUID
454088760 - 455137336 - - - - -
oak-OST0039_UUID
504511856 - 505560432 - - - - -
oak-OST003a_UUID
423460480 - 424509056 - - - - -
oak-OST003b_UUID
558796076 - 559839804 - - - - -
oak-OST003c_UUID
553644448 - 554693024 - - - - -
oak-OST003d_UUID
1543853000 - 1544901576 - - - - -
oak-OST003e_UUID
383811776 - 384860352 - - - - -
oak-OST003f_UUID
421438624 - 422487200 - - - - -
oak-OST0040_UUID
554667372 - 555715948 - - - - -
oak-OST0041_UUID
568768260 - 569816836 - - - - -
oak-OST0042_UUID
639978980 - 641027556 - - - - -
oak-OST0043_UUID
365046548 - 366095124 - - - - -
oak-OST0044_UUID
390849908 - 391898484 - - - - -
oak-OST0045_UUID
963716568 - 964765144 - - - - -
oak-OST0046_UUID
601245336 - 602293912 - - - - -
oak-OST0047_UUID
401730192 - 402778768 - - - - -
oak-OST0048_UUID
528791148 - 529839724 - - - - -
oak-OST0049_UUID
909714900 - 910763476 - - - - -
oak-OST004a_UUID
320610096 - 321658672 - - - - -
oak-OST004b_UUID
467458272 - 468506848 - - - - -
oak-OST004c_UUID
481763152 - 482811728 - - - - -
oak-OST004d_UUID
999019300 - 1000067876 - - - - -
oak-OST004e_UUID
574700184 - 575748760 - - - - -
oak-OST004f_UUID
462401420 - 463449996 - - - - -
oak-OST0050_UUID
481925764 - 482974340 - - - - -
oak-OST0051_UUID
668490020 - 669538596 - - - - -
oak-OST0052_UUID
786108048 - 787156624 - - - - -
oak-OST0053_UUID
511044232 - 512092808 - - - - -
oak-OST0054_UUID
641322280 - 642370856 - - - - -
oak-OST0055_UUID
680894736 - 681943312 - - - - -
oak-OST0056_UUID
523186504 - 524235080 - - - - -
oak-OST0057_UUID
506869632 - 507918208 - - - - -
oak-OST0058_UUID
394409164 - 395457740 - - - - -
oak-OST0059_UUID
991874608 - 992923184 - - - - -
oak-OST005a_UUID
351680568 - 352729144 - - - - -
oak-OST005b_UUID
394870500 - 395919076 - - - - -
oak-OST005c_UUID
333415668 - 334464244 - - - - -
oak-OST005d_UUID
619134820 - 620183396 - - - - -
oak-OST005e_UUID
226679340 - 227727916 - - - - -
oak-OST005f_UUID
299360300 - 300408876 - - - - -
oak-OST0060_UUID
501441284* - 501441284 - - - - -
oak-OST0061_UUID
245723092* - 245723092 - - - - -
oak-OST0062_UUID
280004844 - 280949252 - - - - -
oak-OST0063_UUID
497988600* - 497988600 - - - - -
oak-OST0064_UUID
271022244 - 271581184 - - - - -
oak-OST0065_UUID
750523332* - 750523332 - - - - -
oak-OST0066_UUID
356156588* - 356156588 - - - - -
oak-OST0067_UUID
431379940* - 431379940 - - - - -
oak-OST0068_UUID
423980088* - 423980088 - - - - -
oak-OST0069_UUID
437102468* - 437102468 - - - - -
oak-OST006a_UUID
354793548* - 354793548 - - - - -
oak-OST006b_UUID
527046056* - 527046056 - - - - -
oak-OST006c_UUID
264304292* - 264304292 - - - - -
oak-OST006d_UUID
519332544 - 520381120 - - - - -
oak-OST006e_UUID
413233020* - 413233020 - - - - -
oak-OST006f_UUID
397662008* - 397662008 - - - - -
oak-OST0070_UUID
362951804* - 362951804 - - - - -
oak-OST0071_UUID
705492672* - 705492672 - - - - -
oak-OST0072_UUID
167512940* - 167512940 - - - - -
oak-OST0073_UUID
176038468* - 176038468 - - - - -
oak-OST0074_UUID
338074088* - 338074088 - - - - -
oak-OST0075_UUID
199566968* - 199566968 - - - - -
oak-OST0076_UUID
721559356* - 721559356 - - - - -
oak-OST0077_UUID
395851252 - 396586904 - - - - -
oak-OST0078_UUID
243372096 - 244323420 - - - - -
oak-OST0079_UUID
292555952* - 292555952 - - - - -
oak-OST007a_UUID
322945960* - 322945960 - - - - -
oak-OST007b_UUID
349553564* - 349553564 - - - - -
oak-OST007c_UUID
208454424 - 209437652 - - - - -
oak-OST007d_UUID
194382772* - 194382772 - - - - -
oak-OST007e_UUID
25830772* - 25830772 - - - - -
oak-OST007f_UUID
31056268* - 31056268 - - - - -
oak-OST0080_UUID
63071504* - 63071504 - - - - -
oak-OST0081_UUID
21891996* - 21891996 - - - - -
oak-OST0082_UUID
9831592 - 10880164 - - - - -
oak-OST0083_UUID
29844536* - 29844536 - - - - -
oak-OST0084_UUID
81480528* - 81480528 - - - - -
oak-OST0085_UUID
78325840* - 78325840 - - - - -
oak-OST0086_UUID
18636592* - 18636592 - - - - -
oak-OST0087_UUID
51484216* - 51484216 - - - - -
oak-OST0088_UUID
26217704 - 27045096 - - - - -
oak-OST0089_UUID
17864260* - 17864260 - - - - -
oak-OST008a_UUID
406758852* - 406758852 - - - - -
oak-OST008b_UUID
135026408* - 135026408 - - - - -
oak-OST008c_UUID
136703000* - 136703000 - - - - -
oak-OST008d_UUID
23737664* - 23737664 - - - - -
oak-OST008e_UUID
25915892* - 25915892 - - - - -
oak-OST008f_UUID
27335448* - 27335448 - - - - -
Total allocated inode limit: 10126134, total allocated block limit: 89700409188
I'll report back if I see other quota issues. Thanks again. |
| Comment by Andreas Dilger [ 24/Apr/20 ] |
|
It seems like this is a documentation issue, or maybe the newly-added OSTs need to automatically inherit the "quota.ost=g" setting from the MDT0000 or OST0000 config log when they are first added? |
