https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9671 Lustre <p>Group quotas are not enforced with Lustre 2.9 + patches from <del><a href="https://jira.whamcloud.com/browse/LU-8851" title="nodemap: add flags to limit mapping to UID or GID only" class="issue-link" data-issue-key="LU-8851"><del>LU-8851</del></a></del> (nodemap: add uid/gid only flags to control mapping) and <del><a href="https://jira.whamcloud.com/browse/LU-9258" title="nodemap: group quota ID not properly mapped " class="issue-link" data-issue-key="LU-9258"><del>LU-9258</del></a></del> (nodemap: group quota ID not properly mapped) on servers and regular Lustre 2.9 on clients. This is also a major concern for the upcoming 2.10 release.</p> <p>Group quotas do work OK on clients when using the canonical GID (not mapped).</p> <p>Example follows...</p> <p>Canonical group oak_ruthm GID 3199 is mapped to ruthm (GID 32264) on Sherlock</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@oak-md1-s1 ~]# cat /proc/fs/lustre/nodemap/sherlock/idmap [ ... { idtype: gid, client_id: 32264, fs_id: 3199 } ... ] </pre> </div></div> <p>Client NOT mapped:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@oak-rbh01 ~]# lfs quota -v -g oak_ruthm /oak Disk quotas <span class="code-keyword">for</span> group oak_ruthm (gid 3199): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@oak-rbh01 sthiell]$ id uid=282232(sthiell) gid=3199(oak_ruthm) groups=... [sthiell@oak-rbh01 sthiell]$ dd <span class="code-keyword">if</span>=/dev/zero of=dumpQuota bs=1M dd: error writing ‘dumpQuota’: Disk quota exceeded 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00370499 s, 0.0 kB/s </pre> </div></div> <p>On a mapped Sherlock client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[sthiell@sh-ln01 login! ~]$ lfs quota -v -g ruthm /oak Disk quotas <span class="code-keyword">for</span> group ruthm (gid 32264): Filesystem kbytes quota limit grace files quota limit grace /oak 12629074820* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 384979680 - 384980704 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12691264256 [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ id uid=282232(sthiell) gid=32264(ruthm) groups=... [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ dd <span class="code-keyword">if</span>=/dev/zero of=dumpQuota bs=1M ^C17978+0 records in 17978+0 records out 18851299328 bytes (19 GB) copied, 69.0074 s, 273 MB/s [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ ls -l dumpQuota -rw-rw----+ 1 sthiell ruthm 18851299328 Jun 15 10:56 dumpQuota [sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs getstripe dumpQuota dumpQuota lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 16 obdidx objid objid group 16 2391822 0x247f0e 0 </pre> </div></div> <p>Running lfs quota again shows that the quota limit for oak-OST0010 and also the total allocated block limit have both increased:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[sthiell@sh-ln01 login! /oak/stanford/groups/ruthm/sthiell]$ lfs quota -v -g ruthm /oak Disk quotas <span class="code-keyword">for</span> group ruthm (gid 32264): Filesystem kbytes quota limit grace files quota limit grace /oak 12647484384* 10000000000 10737418240 - 337038* 300000 310000 - oak-MDT0000_UUID 923316 - 0 - 337038 - 338053 - oak-OST0000_UUID 1393407948 - 1393408972 - - - - - oak-OST0001_UUID 1370876152 - 1370879672 - - - - - oak-OST0002_UUID 422046724 - 422047748 - - - - - oak-OST0003_UUID 473785232 - 536870912 - - - - - oak-OST0004_UUID 392781988 - 392783012 - - - - - oak-OST0005_UUID 621784652 - 621785676 - - - - - oak-OST0006_UUID 633895412* - 633895412 - - - - - oak-OST0007_UUID 496028148 - 496029172 - - - - - oak-OST0008_UUID 789069652* - 789069652 - - - - - oak-OST0009_UUID 295415488 - 295416512 - - - - - oak-OST000a_UUID 89056700 - 89057724 - - - - - oak-OST000b_UUID 118396356 - 118397380 - - - - - oak-OST000c_UUID 96884612 - 96885636 - - - - - oak-OST000d_UUID 286116592 - 286117616 - - - - - oak-OST000e_UUID 289566812* - 289566812 - - - - - oak-OST000f_UUID 608782252* - 608782252 - - - - - oak-OST0010_UUID 403389244* - 403389244 - - - - - oak-OST0011_UUID 701799248 - 701800272 - - - - - oak-OST0012_UUID 112087356 - 112088380 - - - - - oak-OST0013_UUID 115159892 - 115160916 - - - - - oak-OST0014_UUID 2325900 - 2326924 - - - - - oak-OST0015_UUID 113333220* - 113333220 - - - - - oak-OST0016_UUID 216462536 - 216463560 - - - - - oak-OST0017_UUID 424851540* - 424851540 - - - - - oak-OST0018_UUID 120107140 - 120108164 - - - - - oak-OST0019_UUID 5978828* - 5978828 - - - - - oak-OST001a_UUID 419459956 - 419460980 - - - - - oak-OST001b_UUID 113303416* - 113303416 - - - - - oak-OST001c_UUID 119018572 - 119019596 - - - - - oak-OST001d_UUID 3682124 - 3683148 - - - - - oak-OST001e_UUID 108871068 - 108872092 - - - - - oak-OST001f_UUID 317494148* - 317494148 - - - - - oak-OST0020_UUID 221257956* - 221257956 - - - - - oak-OST0021_UUID 214119908 - 214120932 - - - - - oak-OST0022_UUID 324983752* - 324983752 - - - - - oak-OST0023_UUID 210980544 - 210981568 - - - - - Total allocated inode limit: 338053, total allocated block limit: 12709672796 </pre> </div></div> <p>Any idea? Happy to provide more inputs if needed. Thanks!</p> <p>Stephane</p> 3.10.0-514.16.1.el7_lustre.x86_64 including fixes for <a href="https://jira.whamcloud.com/browse/LU-8851" title="nodemap: add flags to limit mapping to UID or GID only" class="issue-link" data-issue-key="LU-8851"><strike>LU-8851</strike></a> (nodemap: add uid/gid only flags to control mapping) and <a href="https://jira.whamcloud.com/browse/LU-9258" title="nodemap: group quota ID not properly mapped " class="issue-link" data-issue-key="LU-9258"><strike>LU-9258</strike></a> (nodemap: group quota ID not properly mapped) LU-9671

Group quota not enforced on clients with mapped GID

Bug Critical Resolved Fixed Niu Yawei Stephane Thiell Thu, 15 Jun 2017 18:06:58 +0000 Wed, 15 Dec 2021 02:34:45 +0000 Wed, 19 Jul 2017 03:33:04 +0000 Lustre 2.9.0 Lustre 2.10.1 Lustre 2.11.0 0 4 <p>Niu</p> <p>Could you please advise with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Looks ost write didn't convert back the uid/gid in reply, I'm wondering UID has the same problem. I'll cook a patch soon.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27680" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27680</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9671" title="Group quota not enforced on clients with mapped GID" class="issue-link" data-issue-key="LU-9671"><del>LU-9671</del></a> nodemap: restore client's IDs for OST_WRITE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aeed32066009d737316310cfdce71da672a35988</p> <p>Hi Niu,</p> <p>Thanks so much for your efficient work on this! I noticed that some tests failed though. As soon as the tests pass, I will give this patch a try on our system (I already built updated packages using 2.9-fe as a base without any issue).</p> <p>Best,</p> <p>Stephane</p> <p>Hi Niu,</p> <p>I can also see messages like these on the OSS'es:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0022: cli 845b24d4-9cbd-3aa6-d656-32638dc6d941 claims 1073152 GRANT, real grant 0 Jun 21 03:28:00 oak-io1-s1 kernel: LustreError: 127028:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 10486 previous similar messages </pre> </div></div> <p>... which I assume are related to this issue?<br/> Thanks!<br/> Stephane</p> <p>I don't think the error messages are related to this issue, it indicates something wrong in the grant code, could you please open another ticket for this grant issue? Thanks.</p> <p>Thanks Niu. I opened <a href="https://jira.whamcloud.com/browse/LU-9704" title="ofd_grant_check() claims GRANT, real grant 0" class="issue-link" data-issue-key="LU-9704"><del>LU-9704</del></a> for the grant issue.</p> <p>Hi Niu,</p> <p>We have been running your latest patch from <a href="https://review.whamcloud.com/#/c/27680/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/27680/</a> for a few weeks now without any issue. Quotas are properly enforced. Too bad this missed 2.10 release as without it Lustre quotas are broken. Could Intel land it to master and integrate the patch to 2.10.1?</p> <p>Thanks much!</p> <p>Stephane</p> <p>Stephane</p> <p>Yes it is unfortunate that this fix only became ready to land after code freeze for 2.10.0 but it can certianly be included in 2.10.1</p> <p>Peter</p> <p>Hi Peter,</p> <p>Awesome, thanks Peter!</p> <p>Stephane</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27680/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27680/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9671" title="Group quota not enforced on clients with mapped GID" class="issue-link" data-issue-key="LU-9671"><del>LU-9671</del></a> nodemap: restore client's IDs for OST_WRITE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e207f9f96fc51f3b6d219193cca3d83aaa99b3e8</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28221" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28221</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9671" title="Group quota not enforced on clients with mapped GID" class="issue-link" data-issue-key="LU-9671"><del>LU-9671</del></a> nodemap: restore client's IDs for OST_WRITE<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 470e29d53deb1064b7c4aa3ee99b30cc02b36892</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28221/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28221/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9671" title="Group quota not enforced on clients with mapped GID" class="issue-link" data-issue-key="LU-9671"><del>LU-9671</del></a> nodemap: restore client's IDs for OST_WRITE<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 0bef1fb84d6d93c36db90f92f8f9fc2e68baaa97</p> Related LU-9258 LU-8851 Development Epic/Theme Quota Rank 1|hzzf7z: Rank (Obsolete) 9223372036854775807 Severity