Details
Critical -
3
-
9223372036854775807
Description
Group quotas are not enforced with Lustre 2.9 + patches from LU-8851LU-9258
Group quotas do work OK on clients when using the canonical GID (not mapped).
Example follows...
Canonical group oak_ruthm GID 3199 is mapped to ruthm (GID 32264) on Sherlock
[root@oak-md1-s1 ~]# cat /proc/fs/lustre/nodemap/sherlock/idmap
[
... { idtype: gid, client_id: 32264, fs_id: 3199 } ...
]
Client NOT mapped:
[root@oak-rbh01 ~]# lfs quota -v -g oak_ruthm /oak Disk quotas for group oak_ruthm (gid 3199): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@oak-rbh01 sthiell]$ id uid=282232(sthiell) gid=3199(oak_ruthm) groups=... [sthiell@oak-rbh01 sthiell]$ dd if=/dev/zero of=dumpQuota bs=1M dd: error writing ‘dumpQuota’: Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00370499 s, 0.0 kB/s
On a mapped Sherlock client:
[sthiell@sh-ln01 login! ~]$ lfs quota -v -g ruthm /oak Disk quotas for group ruthm (gid 32264): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ id uid=282232(sthiell) gid=32264(ruthm) groups=... [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ dd if=/dev/zero of=dumpQuota bs=1M ^C17978+0 records in 17978+0 records out 18851299328 bytes (19 GB) copied, 69.0074 s, 273 MB/s [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ ls -l dumpQuota -rw-rw----+ 1 sthiell ruthm 18851299328 Jun 15 10:56 dumpQuota [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs getstripe dumpQuota dumpQuota lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 16 obdidx objid objid group 16 2391822 0x247f0e 0
Running lfs quota again shows that the quota limit for oak-OST0010 and also the total allocated block limit have both increased:
[sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs quota -v -g ruthm /oak
Disk quotas for group ruthm (gid 32264):
Filesystem kbytes quota limit grace files quota limit grace
/oak 12647484384* 10000000000 10737418240 - 337038* 300000 310000 -
oak-MDT0000_UUID
923316 - 0 - 337038 - 338053 -
oak-OST0000_UUID
1393407948 - 1393408972 - - - - -
oak-OST0001_UUID
1370876152 - 1370879672 - - - - -
oak-OST0002_UUID
422046724 - 422047748 - - - - -
oak-OST0003_UUID
473785232 - 536870912 - - - - -
oak-OST0004_UUID
392781988 - 392783012 - - - - -
oak-OST0005_UUID
621784652 - 621785676 - - - - -
oak-OST0006_UUID
633895412* - 633895412 - - - - -
oak-OST0007_UUID
496028148 - 496029172 - - - - -
oak-OST0008_UUID
789069652* - 789069652 - - - - -
oak-OST0009_UUID
295415488 - 295416512 - - - - -
oak-OST000a_UUID
89056700 - 89057724 - - - - -
oak-OST000b_UUID
118396356 - 118397380 - - - - -
oak-OST000c_UUID
96884612 - 96885636 - - - - -
oak-OST000d_UUID
286116592 - 286117616 - - - - -
oak-OST000e_UUID
289566812* - 289566812 - - - - -
oak-OST000f_UUID
608782252* - 608782252 - - - - -
oak-OST0010_UUID
403389244* - 403389244 - - - - -
oak-OST0011_UUID
701799248 - 701800272 - - - - -
oak-OST0012_UUID
112087356 - 112088380 - - - - -
oak-OST0013_UUID
115159892 - 115160916 - - - - -
oak-OST0014_UUID
2325900 - 2326924 - - - - -
oak-OST0015_UUID
113333220* - 113333220 - - - - -
oak-OST0016_UUID
216462536 - 216463560 - - - - -
oak-OST0017_UUID
424851540* - 424851540 - - - - -
oak-OST0018_UUID
120107140 - 120108164 - - - - -
oak-OST0019_UUID
5978828* - 5978828 - - - - -
oak-OST001a_UUID
419459956 - 419460980 - - - - -
oak-OST001b_UUID
113303416* - 113303416 - - - - -
oak-OST001c_UUID
119018572 - 119019596 - - - - -
oak-OST001d_UUID
3682124 - 3683148 - - - - -
oak-OST001e_UUID
108871068 - 108872092 - - - - -
oak-OST001f_UUID
317494148* - 317494148 - - - - -
oak-OST0020_UUID
221257956* - 221257956 - - - - -
oak-OST0021_UUID
214119908 - 214120932 - - - - -
oak-OST0022_UUID
324983752* - 324983752 - - - - -
oak-OST0023_UUID
210980544 - 210981568 - - - - -
Total allocated inode limit: 338053, total allocated block limit: 12709672796
Any idea? Happy to provide more inputs if needed. Thanks!
Stephane
Attachments
Issue Links
Activity
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/28221
Subject: LU-9671 nodemap: restore client's IDs for OST_WRITE
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 470e29d53deb1064b7c4aa3ee99b30cc02b36892
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/27680/
Subject: LU-9671 nodemap: restore client's IDs for OST_WRITE
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e207f9f96fc51f3b6d219193cca3d83aaa99b3e8
Stephane
Yes it is unfortunate that this fix only became ready to land after code freeze for 2.10.0 but it can certianly be included in 2.10.1
Peter
Hi Niu,
We have been running your latest patch from https://review.whamcloud.com/#/c/27680/ for a few weeks now without any issue. Quotas are properly enforced. Too bad this missed 2.10 release as without it Lustre quotas are broken. Could Intel land it to master and integrate the patch to 2.10.1?
Thanks much!
Stephane
I don't think the error messages are related to this issue, it indicates something wrong in the grant code, could you please open another ticket for this grant issue? Thanks.
Hi Niu,
I can also see messages like these on the OSS'es:
Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0022: cli 845b24d4-9cbd-3aa6-d656-32638dc6d941 claims 1073152 GRANT, real grant 0 Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 10486 previous similar messages
... which I assume are related to this issue?
Thanks!
Stephane
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/28221/
Subject:
LU-9671nodemap: restore client's IDs for OST_WRITEProject: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 0bef1fb84d6d93c36db90f92f8f9fc2e68baaa97