Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.10.1
-
None
-
CentOS 7.4
-
3
-
9223372036854775807
Description
Hi,
We are seeing quota problems with 2.10.1 where, from time to time, group quotas are generating EDQUOT (users are actually reporting the problem) while there is room left. Some OSTs are seen as full as shown below and the rebalancing doesn't seem to work:
[root@oak-rbh01 ~]# lfs quota -v -g oak_p-cvmed /oak
Disk quotas for grp oak_p-cvmed (gid 3683):
Filesystem kbytes quota limit grace files quota limit grace
/oak 96389875036 120000000000 120000000000 - 6955998 18000000 18000000 -
oak-MDT0000_UUID
3248820 - 0 - 6955998 - 8388608 -
oak-OST0000_UUID
1196660392 - 1342177280 - - - - -
oak-OST0001_UUID
1908235316 - 2147483648 - - - - -
oak-OST0002_UUID
1225832336 - 1342177280 - - - - -
oak-OST0003_UUID
1327591244 - 1342177280 - - - - -
oak-OST0004_UUID
1965895968 - 2147483648 - - - - -
oak-OST0005_UUID
1159391612 - 1342177280 - - - - -
oak-OST0006_UUID
1635255040 - 1879048192 - - - - -
oak-OST0007_UUID
1818596964 - 1879048192 - - - - -
oak-OST0008_UUID
1872031764 - 1879048192 - - - - -
oak-OST0009_UUID
2061279604 - 2147483648 - - - - -
oak-OST000a_UUID
1445543488 - 1610612736 - - - - -
oak-OST000b_UUID
1875314700 - 1879048192 - - - - -
oak-OST000c_UUID
1301881412 - 1342177280 - - - - -
oak-OST000d_UUID
1766688092 - 1879048192 - - - - -
oak-OST000e_UUID
2005981712 - 2147483648 - - - - -
oak-OST000f_UUID
1491138396 - 1610612736 - - - - -
oak-OST0010_UUID
1292096088 - 1342177280 - - - - -
oak-OST0011_UUID
1222866272 - 1342177280 - - - - -
oak-OST0012_UUID
1312869104 - 1342177280 - - - - -
oak-OST0013_UUID
1185445504 - 1342177280 - - - - -
oak-OST0014_UUID
1315544800 - 1342177280 - - - - -
oak-OST0015_UUID
2025717256 - 2147483648 - - - - -
oak-OST0016_UUID
1817010800 - 1879048192 - - - - -
oak-OST0017_UUID
1699092560 - 1879048192 - - - - -
oak-OST0018_UUID
1921966992 - 2147483648 - - - - -
oak-OST0019_UUID
1752975104 - 1879048192 - - - - -
oak-OST001a_UUID
2022449576 - 2147483648 - - - - -
oak-OST001b_UUID
1476019956 - 1610612736 - - - - -
oak-OST001c_UUID
2002420900 - 2147483648 - - - - -
oak-OST001d_UUID
1175776272 - 1342177280 - - - - -
oak-OST001e_UUID
1522667428 - 1610612736 - - - - -
oak-OST001f_UUID
1698940868 - 1879048192 - - - - -
oak-OST0020_UUID
1418438600 - 1610612736 - - - - -
oak-OST0021_UUID
1848558676 - 1879048192 - - - - -
oak-OST0022_UUID
1567670312 - 1610612736 - - - - -
oak-OST0023_UUID
1755882404 - 1879048192 - - - - -
oak-OST0024_UUID
1725770704 - 1879048192 - - - - -
oak-OST0025_UUID
2021496552 - 2147483648 - - - - -
oak-OST0026_UUID
2340218652 - 2415919104 - - - - -
oak-OST0027_UUID
2078849960 - 2147483648 - - - - -
oak-OST0028_UUID
2401223300 - 2415919104 - - - - -
oak-OST0029_UUID
2255153880 - 2415919104 - - - - -
oak-OST002a_UUID
2479360100 - 2684354560 - - - - -
oak-OST002b_UUID
1956889380 - 2147483648 - - - - -
oak-OST002c_UUID
2336034612 - 2415919104 - - - - -
oak-OST002d_UUID
1897045500 - 2147483648 - - - - -
oak-OST002e_UUID
2069066412 - 2147483648 - - - - -
oak-OST002f_UUID
2668099124 - 2684354560 - - - - -
oak-OST0030_UUID
302970856 - 536870912 - - - - -
oak-OST0031_UUID
425767268 - 536870912 - - - - -
oak-OST0032_UUID
554265344 - 805306368 - - - - -
oak-OST0033_UUID
616158116 - 805306368 - - - - -
oak-OST0034_UUID
523406904 - 536870912 - - - - -
oak-OST0035_UUID
832949332 - 1073741824 - - - - -
oak-OST0036_UUID
431649588 - 536870912 - - - - -
oak-OST0037_UUID
335297304 - 536870912 - - - - -
oak-OST0038_UUID
768953372 - 805306368 - - - - -
oak-OST0039_UUID
589398720 - 805306368 - - - - -
oak-OST003a_UUID
822149664 - 1073741824 - - - - -
oak-OST003b_UUID
246038976 - 268435456 - - - - -
oak-OST003c_UUID
1002757608 - 1073741824 - - - - -
oak-OST003d_UUID
655190956 - 805306368 - - - - -
oak-OST003e_UUID
464755608* - 464755608 - - - - -
oak-OST003f_UUID
265537376 - 268435456 - - - - -
oak-OST0040_UUID
380491764 - 536870912 - - - - -
oak-OST0041_UUID
628194908 - 805306368 - - - - -
oak-OST0042_UUID
220394524 - 268435456 - - - - -
oak-OST0043_UUID
388284936 - 536870912 - - - - -
oak-OST0044_UUID
429979492 - 536870912 - - - - -
oak-OST0045_UUID
276764380 - 536870912 - - - - -
oak-OST0046_UUID
346999308 - 536870912 - - - - -
oak-OST0047_UUID
408032656 - 536870912 - - - - -
oak-OST0048_UUID
17760704 - 268435456 - - - - -
oak-OST0049_UUID
11267016 - 268435456 - - - - -
oak-OST004a_UUID
12111344 - 268435456 - - - - -
oak-OST004b_UUID
8750240 - 268435456 - - - - -
oak-OST004c_UUID
31825656 - 268435456 - - - - -
oak-OST004d_UUID
67586608 - 268435456 - - - - -
Total allocated inode limit: 8388608, total allocated block limit: 106765196184
In some cases I was able to disable/enable group q uota to fix the problem but in this particular, I can't find a way to force a refresh. Any idea would be welcome? This has a pretty important impact on some groups.
Thanks!
Stephane