[LU-9671] Group quota not enforced on clients with mapped GID Created: 15/Jun/17 Updated: 15/Dec/21 Resolved: 19/Jul/17 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.9.0 |
| Fix Version/s: | Lustre 2.10.1, Lustre 2.11.0 |
| Type: | Bug | Priority: | Critical |
| Reporter: | Stephane Thiell | Assignee: | Niu Yawei (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
3.10.0-514.16.1.el7_lustre.x86_64 including fixes for |
||
| Issue Links: |
|
||||||||||||
| Epic/Theme: | Quota | ||||||||||||
| Severity: | 3 | ||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||
| Description |
|
Group quotas are not enforced with Lustre 2.9 + patches from Group quotas do work OK on clients when using the canonical GID (not mapped). Example follows... Canonical group oak_ruthm GID 3199 is mapped to ruthm (GID 32264) on Sherlock [root@oak-md1-s1 ~]# cat /proc/fs/lustre/nodemap/sherlock/idmap
[
... { idtype: gid, client_id: 32264, fs_id: 3199 } ...
]
Client NOT mapped: [root@oak-rbh01 ~]# lfs quota -v -g oak_ruthm /oak Disk quotas for group oak_ruthm (gid 3199): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@oak-rbh01 sthiell]$ id uid=282232(sthiell) gid=3199(oak_ruthm) groups=... [sthiell@oak-rbh01 sthiell]$ dd if=/dev/zero of=dumpQuota bs=1M dd: error writing ‘dumpQuota’: Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00370499 s, 0.0 kB/s On a mapped Sherlock client: [sthiell@sh-ln01 login! ~]$ lfs quota -v -g ruthm /oak Disk quotas for group ruthm (gid 32264): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ id uid=282232(sthiell) gid=32264(ruthm) groups=... [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ dd if=/dev/zero of=dumpQuota bs=1M ^C17978+0 records in 17978+0 records out 18851299328 bytes (19 GB) copied, 69.0074 s, 273 MB/s [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ ls -l dumpQuota -rw-rw----+ 1 sthiell ruthm 18851299328 Jun 15 10:56 dumpQuota [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs getstripe dumpQuota dumpQuota lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 16 obdidx objid objid group 16 2391822 0x247f0e 0 Running lfs quota again shows that the quota limit for oak-OST0010 and also the total allocated block limit have both increased: [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs quota -v -g ruthm /oak
Disk quotas for group ruthm (gid 32264):
Filesystem kbytes quota limit grace files quota limit grace
/oak 12647484384* 10000000000 10737418240 - 337038* 300000 310000 -
oak-MDT0000_UUID
923316 - 0 - 337038 - 338053 -
oak-OST0000_UUID
1393407948 - 1393408972 - - - - -
oak-OST0001_UUID
1370876152 - 1370879672 - - - - -
oak-OST0002_UUID
422046724 - 422047748 - - - - -
oak-OST0003_UUID
473785232 - 536870912 - - - - -
oak-OST0004_UUID
392781988 - 392783012 - - - - -
oak-OST0005_UUID
621784652 - 621785676 - - - - -
oak-OST0006_UUID
633895412* - 633895412 - - - - -
oak-OST0007_UUID
496028148 - 496029172 - - - - -
oak-OST0008_UUID
789069652* - 789069652 - - - - -
oak-OST0009_UUID
295415488 - 295416512 - - - - -
oak-OST000a_UUID
89056700 - 89057724 - - - - -
oak-OST000b_UUID
118396356 - 118397380 - - - - -
oak-OST000c_UUID
96884612 - 96885636 - - - - -
oak-OST000d_UUID
286116592 - 286117616 - - - - -
oak-OST000e_UUID
289566812* - 289566812 - - - - -
oak-OST000f_UUID
608782252* - 608782252 - - - - -
oak-OST0010_UUID
403389244* - 403389244 - - - - -
oak-OST0011_UUID
701799248 - 701800272 - - - - -
oak-OST0012_UUID
112087356 - 112088380 - - - - -
oak-OST0013_UUID
115159892 - 115160916 - - - - -
oak-OST0014_UUID
2325900 - 2326924 - - - - -
oak-OST0015_UUID
113333220* - 113333220 - - - - -
oak-OST0016_UUID
216462536 - 216463560 - - - - -
oak-OST0017_UUID
424851540* - 424851540 - - - - -
oak-OST0018_UUID
120107140 - 120108164 - - - - -
oak-OST0019_UUID
5978828* - 5978828 - - - - -
oak-OST001a_UUID
419459956 - 419460980 - - - - -
oak-OST001b_UUID
113303416* - 113303416 - - - - -
oak-OST001c_UUID
119018572 - 119019596 - - - - -
oak-OST001d_UUID
3682124 - 3683148 - - - - -
oak-OST001e_UUID
108871068 - 108872092 - - - - -
oak-OST001f_UUID
317494148* - 317494148 - - - - -
oak-OST0020_UUID
221257956* - 221257956 - - - - -
oak-OST0021_UUID
214119908 - 214120932 - - - - -
oak-OST0022_UUID
324983752* - 324983752 - - - - -
oak-OST0023_UUID
210980544 - 210981568 - - - - -
Total allocated inode limit: 338053, total allocated block limit: 12709672796
Any idea? Happy to provide more inputs if needed. Thanks! Stephane |
| Comments |
| Comment by Peter Jones [ 15/Jun/17 ] |
|
Niu Could you please advise with this one? Thanks Peter |
| Comment by Niu Yawei (Inactive) [ 16/Jun/17 ] |
|
Looks ost write didn't convert back the uid/gid in reply, I'm wondering UID has the same problem. I'll cook a patch soon. |
| Comment by Gerrit Updater [ 16/Jun/17 ] |
|
Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/27680 |
| Comment by Stephane Thiell [ 16/Jun/17 ] |
|
Hi Niu, Thanks so much for your efficient work on this! I noticed that some tests failed though. As soon as the tests pass, I will give this patch a try on our system (I already built updated packages using 2.9-fe as a base without any issue). Best, Stephane |
| Comment by Stephane Thiell [ 22/Jun/17 ] |
|
Hi Niu, I can also see messages like these on the OSS'es: Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0022: cli 845b24d4-9cbd-3aa6-d656-32638dc6d941 claims 1073152 GRANT, real grant 0 Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 10486 previous similar messages ... which I assume are related to this issue? |
| Comment by Niu Yawei (Inactive) [ 22/Jun/17 ] |
|
I don't think the error messages are related to this issue, it indicates something wrong in the grant code, could you please open another ticket for this grant issue? Thanks. |
| Comment by Stephane Thiell [ 22/Jun/17 ] |
|
Thanks Niu. I opened |
| Comment by Stephane Thiell [ 14/Jul/17 ] |
|
Hi Niu, We have been running your latest patch from https://review.whamcloud.com/#/c/27680/ for a few weeks now without any issue. Quotas are properly enforced. Too bad this missed 2.10 release as without it Lustre quotas are broken. Could Intel land it to master and integrate the patch to 2.10.1? Thanks much! Stephane |
| Comment by Peter Jones [ 14/Jul/17 ] |
|
Stephane Yes it is unfortunate that this fix only became ready to land after code freeze for 2.10.0 but it can certianly be included in 2.10.1 Peter |
| Comment by Stephane Thiell [ 14/Jul/17 ] |
|
Hi Peter, Awesome, thanks Peter! Stephane |
| Comment by Gerrit Updater [ 19/Jul/17 ] |
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/27680/ |
| Comment by Peter Jones [ 19/Jul/17 ] |
|
Landed for 2.11 |
| Comment by Gerrit Updater [ 26/Jul/17 ] |
|
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/28221 |
| Comment by Gerrit Updater [ 03/Aug/17 ] |
|
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/28221/ |