https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12500 Lustre <p>Hi,</p> <p>it doesn't look like lustre is enforcing group block quota. eg.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> &gt; lfs quota -g oz011 /fred Disk quotas for grp oz011 (gid 10206): Filesystem kbytes quota limit grace files quota limit grace /fred 10893023318* 0 10737418240 - 57475 0 1000000 - &gt; dd if=/tmp/urand100 of=/fred/oz011/blah bs=1M 1000+1 records in 1000+1 records out 1048738400 bytes (1.0 GB) copied, 0.995716 s, 1.1 GB/s &gt; ls -lsh /fred/oz011/blah 688M -rw-r--r-- 1 user oz011 1001M Jul 1 21:42 /fred/oz011/blah &gt; lfs quota -g oz011 /fred Disk quotas for grp oz011 (gid 10206): Filesystem kbytes quota limit grace files quota limit grace /fred 10894895294* 0 10737418240 - 57477 0 1000000 - </pre> </div></div> <p>I can see old quota bugs that look similar, but none currently open.</p> <p>all our directories are setgid. ie.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> &gt; ls -ld /fred/oz011 drwxrws--- 17 root oz011 33280 Jul 1 21:42 /fred/oz011 </pre> </div></div> <p>server is 2.10.5 plus these</p> <p>lu11082-lu11103-stuckMdtThreads-gerrit32853-3dc08caa.diff<br/> lu11418-refreshStale-gerrit33401-v4-71f409c9.diff<br/> lu11111-lfsck-gerrit32796-693fe452.ported.patch<br/> lu11418-stopOrphCleanupDaThreadSpinning-gerrit33662-45434fd0.diff<br/> lu11201-lfsckDoesntFinish-gerrit33078-4829fb05.patch<br/> lu11419-lfsckDoesntFinish-gerrit33252-22503a1d.diff<br/> lu11301-stuckMdtThreads2-c43baa1c.patch<br/> lu11663-partialPageCorruption-gerrit33748-18d6b8fb.diff<br/> lu11418-hungMdtZfs-gerrit33248-eaa3c60d.diff</p> <p>not all of which are in 2.10.x AFAIK (all but one are in 2.12?), so it'd unfortunately be quite a bit of work to update servers to 2.10.8.</p> <p>clients are all stock 2.10.7</p> <p>thanks</p> <p>cheers,<br/> robin</p> 2.10.5 + lots of patches on servers, x86_64, zfs 0.7.9, OPA <br/> 2.10.7 on clients, x86_64, OPA <br/> group block and inode quotas on. <br/> LU-12500

group block quota limits not enforced

Bug Minor Resolved Not a Bug Hongchao Zhang SC Admin Mon, 1 Jul 2019 15:57:03 +0000 Mon, 1 Jul 2019 18:41:30 +0000 Mon, 1 Jul 2019 18:41:30 +0000 Lustre 2.10.5 0 4 <p>a bit more info is that if I wait a while then the "size on disk" of the file increases. ie. now it's</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> &gt; ls -lsh /fred/oz011/blah 915M -rw-r--r-- 1 user oz011 1001M Jul 1 21:42 /fred/oz011/blah </pre> </div></div> <p>perhaps that's normal. I forget... :-/</p> <p>a 3rd bit of info is that I suspect this quota issue also causes unusually high load on OSS's. we recently had a user in (what we now realise was) an over-quota group running 300+ i/o intensive jobs. this caused load on OSS's of 200+. one OSS was STONITH'd because it hit a timeout - probably just from the load.</p> <p>cheers,<br/> robin</p> <p>Hongchao</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>It would be good to check that you've got quota <b>enforcement</b> enabled, and not just accounting.</p> <p>This is described in the quota section of the Lustre operations manual, but you can check with this command on the MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl get_param osd-*.*.quota_slave.info </pre> </div></div> <p>If you do not have both 'u' and 'g' under 'enabled', then quota enforcement is not enabled.</p> <p>ah. a beer for Patrick <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> # cexec -p warble:2 oss:1-10 'lctl get_param osd-*.*.quota_slave.info' | grep enable warble warble2: quota enabled: g warble warble2: quota enabled: g warble warble2: quota enabled: g oss arkle1: quota enabled: g oss arkle1: quota enabled: g oss arkle2: quota enabled: g oss arkle2: quota enabled: g oss arkle3: quota enabled: g oss arkle3: quota enabled: g oss arkle4: quota enabled: g oss arkle4: quota enabled: g oss arkle5: quota enabled: g oss arkle5: quota enabled: g oss arkle6: quota enabled: g oss arkle6: quota enabled: g oss arkle7: quota enabled: g oss arkle7: quota enabled: g oss arkle8: quota enabled: g oss arkle8: quota enabled: g oss arkle9: quota enabled: none oss arkle9: quota enabled: none oss arkle10: quota enabled: none oss arkle10: quota enabled: none </pre> </div></div> <p>we added 2 more OSS's a while back, but looks like the conf_param isn't inherited to new OSS's.</p> <p>is that expected behaviour?</p> <p>I re-did the conf_param on the MGS</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[warble1]root: lctl conf_param dagg.quota.ost=g </pre> </div></div> <p>and now it looks like dd's still writes some data, but not much, and at least there's an error coming back too -&gt;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> &gt; dd if=/tmp/urand100 of=/fred/oz011/blah bs=1M dd: error writing '/fred/oz011/blah': Disk quota exceeded 10+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.404197 s, 23.3 MB/s &gt; dd if=/tmp/urand100 of=/fred/oz011/blah2 bs=1M dd: error writing '/fred/oz011/blah2': Disk quota exceeded 9+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.199364 s, 42.1 MB/s </pre> </div></div> <p>so that's probably fine.</p> <p>if this is all expected behaviour then please close this ticket. thanks!</p> <p>cheers,<br/> robin</p> <p>Yeah, this is not ideal behavior (re: inheritance with the new OSTs), but it is expected.</p> <p>Glad to be of assistance!</p> <p>Config issue at customer site.</p> Development Rank 1|i00j1z: Rank (Obsolete) 9223372036854775807 Severity