https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13043 Lustre <p>We're performing the last tests before putting Project quotas in production on Sherlock/Fir. We noticed today that when using enforced project quotas, with a default quota limits set, we can see a lot of error messages on all OSS when a user reaches the block quota limit (not the inode limit):</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50958:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 112944:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50909:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 51093:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43806:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43917:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 105082:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 55797:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 62343:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 84857:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 120934:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58191:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58092:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 109202:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 48652:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33426:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33422:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33427:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74530:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74518:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74519:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33395:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74482:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:44 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 66898:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 3759:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:53 fir-io8-s1 kernel: LustreError: 105052:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:58 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:36:03 fir-io6-s1 kernel: LustreError: 48651:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:36:03 fir-io4-s2 kernel: LustreError: 74524:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:36:03 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 </pre> </div></div> <p>Block and inode quota limits are properly enforced, we're just curious to know how we can disable these messages.</p> <p>This is the output of lfs quota when reaching the block quota limits and the messages appear on the servers:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[sthiell@sh-ln07 login /scratch/users/sthiell/sink]$ lfs quota -vp $(id -u) /scratch Disk quotas for prj 282232 (pid 282232): Filesystem kbytes quota limit grace files quota limit grace /scratch 45001977720* 45000000000 45000000000 - 20409287 30000000 30000000 - fir-MDT0000_UUID 10257956* - 10257956 - 20409287 - 20764938 - fir-MDT0001_UUID 0 - 4 - 0 - 1 - fir-MDT0002_UUID 0 - 13753024 - 0 - 4194304 - fir-MDT0003_UUID 0 - 4 - 0 - 1 - fir-OST0000_UUID 464361544* - 464361544 - - - - - fir-OST0001_UUID 559676264* - 559676264 - - - - - fir-OST0002_UUID 507091492 - 507092496 - - - - - fir-OST0003_UUID 392618344* - 392618344 - - - - - fir-OST0004_UUID 942404472 - 942404480 - - - - - fir-OST0005_UUID 449001624* - 449001624 - - - - - fir-OST0006_UUID 397667212* - 397667212 - - - - - fir-OST0007_UUID 490947300* - 490947300 - - - - - fir-OST0008_UUID 465962552 - 465962592 - - - - - fir-OST0009_UUID 391662704* - 391662704 - - - - - fir-OST000a_UUID 448462700* - 448462700 - - - - - fir-OST000b_UUID 285858668* - 285858668 - - - - - fir-OST000c_UUID 452055424* - 452055424 - - - - - fir-OST000d_UUID 664806004* - 664806004 - - - - - fir-OST000e_UUID 435402460* - 435402460 - - - - - fir-OST000f_UUID 357099424* - 357099424 - - - - - fir-OST0010_UUID 571025464* - 571025464 - - - - - fir-OST0011_UUID 476045756* - 476045756 - - - - - fir-OST0012_UUID 401734136* - 401734136 - - - - - fir-OST0013_UUID 563884544* - 563884544 - - - - - fir-OST0014_UUID 451027312 - 451028316 - - - - - fir-OST0015_UUID 1008410220* - 1008410220 - - - - - fir-OST0016_UUID 459554296* - 459554296 - - - - - fir-OST0017_UUID 406426536 - 406427548 - - - - - fir-OST0018_UUID 431504956 - 431505964 - - - - - fir-OST0019_UUID 488622276* - 488622276 - - - - - fir-OST001a_UUID 374191672* - 374191672 - - - - - fir-OST001b_UUID 468069876 - 468069908 - - - - - fir-OST001c_UUID 488161748* - 488161748 - - - - - fir-OST001d_UUID 500656704* - 500656704 - - - - - fir-OST001e_UUID 552511176* - 552511176 - - - - - fir-OST001f_UUID 500188684* - 500188684 - - - - - fir-OST0020_UUID 356752432* - 356752432 - - - - - fir-OST0021_UUID 514022332* - 514022332 - - - - - fir-OST0022_UUID 388244648 - 388245656 - - - - - fir-OST0023_UUID 478265396* - 478265396 - - - - - fir-OST0024_UUID 662203064 - 662204080 - - - - - fir-OST0025_UUID 357218288 - 357219296 - - - - - fir-OST0026_UUID 388994408* - 388994408 - - - - - fir-OST0027_UUID 416523512* - 416523512 - - - - - fir-OST0028_UUID 308196948* - 308196948 - - - - - fir-OST0029_UUID 387409384* - 387409384 - - - - - fir-OST002a_UUID 449330736* - 449330736 - - - - - fir-OST002b_UUID 429416836* - 429416836 - - - - - fir-OST002c_UUID 394779272* - 394779272 - - - - - fir-OST002d_UUID 452150536* - 452150536 - - - - - fir-OST002e_UUID 649498168 - 649499192 - - - - - fir-OST002f_UUID 607361816* - 607361816 - - - - - fir-OST0030_UUID 214663440* - 214663440 - - - - - fir-OST0031_UUID 420568284* - 420568284 - - - - - fir-OST0032_UUID 380047312 - 380048316 - - - - - fir-OST0033_UUID 1057671372* - 1057671372 - - - - - fir-OST0034_UUID 996787540 - 996788524 - - - - - fir-OST0035_UUID 1088752228* - 1088752228 - - - - - fir-OST0036_UUID 481519020* - 481519020 - - - - - fir-OST0037_UUID 323877700* - 323877700 - - - - - fir-OST0038_UUID 272179672* - 272179672 - - - - - fir-OST0039_UUID 475962020* - 475962020 - - - - - fir-OST003a_UUID 291771688* - 291771688 - - - - - fir-OST003b_UUID 248632172* - 248632172 - - - - - fir-OST003c_UUID 632507520* - 632507520 - - - - - fir-OST003d_UUID 383028632* - 383028632 - - - - - fir-OST003e_UUID 355118096* - 355118096 - - - - - fir-OST003f_UUID 787302308* - 787302308 - - - - - fir-OST0040_UUID 330605936 - 330606928 - - - - - fir-OST0041_UUID 453859240* - 453859240 - - - - - fir-OST0042_UUID 440181540* - 440181540 - - - - - fir-OST0043_UUID 261950232* - 261950232 - - - - - fir-OST0044_UUID 216148944* - 216148944 - - - - - fir-OST0045_UUID 416328968* - 416328968 - - - - - fir-OST0046_UUID 173872668* - 173872668 - - - - - fir-OST0047_UUID 852739020 - 852740028 - - - - - fir-OST0048_UUID 377475652* - 377475652 - - - - - fir-OST0049_UUID 265111876 - 265111908 - - - - - fir-OST004a_UUID 400281672* - 400281672 - - - - - fir-OST004b_UUID 414808608* - 414808608 - - - - - fir-OST004c_UUID 138487876* - 138487876 - - - - - fir-OST004d_UUID 564593396* - 564593396 - - - - - fir-OST004e_UUID 218610204* - 218610204 - - - - - fir-OST004f_UUID 413015672* - 413015672 - - - - - fir-OST0050_UUID 508523192 - 508524216 - - - - - fir-OST0051_UUID 581310148* - 581310148 - - - - - fir-OST0052_UUID 198038980* - 198038980 - - - - - fir-OST0053_UUID 360050332* - 360050332 - - - - - fir-OST0054_UUID 229448744* - 229448744 - - - - - fir-OST0055_UUID 559147340* - 559147340 - - - - - fir-OST0056_UUID 532999060* - 532999060 - - - - - fir-OST0057_UUID 339694332 - 339695352 - - - - - fir-OST0058_UUID 986951040* - 986951040 - - - - - fir-OST0059_UUID 175977116* - 175977116 - - - - - fir-OST005a_UUID 372989596* - 372989596 - - - - - fir-OST005b_UUID 662420908* - 662420908 - - - - - fir-OST005c_UUID 445986612* - 445986612 - - - - - fir-OST005d_UUID 302969960* - 302969960 - - - - - fir-OST005e_UUID 453619292* - 453619292 - - - - - fir-OST005f_UUID 845671284* - 845671284 - - - - - Total allocated inode limit: 24959244, total allocated block limit: 44991733992 pid 282232 is using default block quota setting pid 282232 is using default file quota setting </pre> </div></div> CentOS 7.6 / 2.12.3 servers and clients LU-13043

Project quotas: force to ignore quota flags =8 on quota exceeded

Bug Minor Resolved Fixed Wang Shilong Stephane Thiell Mon, 2 Dec 2019 23:52:41 +0000 Fri, 10 Jan 2020 12:56:00 +0000 Fri, 3 Jan 2020 02:41:28 +0000 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.4 0 6 <p>Shilong</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Looking into this quickly, it appears that "<tt>QUOTA_FL_OVER_PRJQUOTA = 8</tt>" but this does not have a matching <tt>OSD_QID_PRJ</tt> definition in <tt>enum osd_qid_declare_flags</tt>. Using "<tt>CDEBUG(D_ERROR, ...)</tt>" for this message is bad, since this prints to the console and does not rate limit the messages. At a very minimum, this should be changed to "<tt>CERROR()</tt>" so the message is rate limited, but I question whether this message is needed on the console at all. The admin shouldn't be getting console error messages when a user goes over quota (this would be happening continuously at some sites), so this would probably be better as "<tt>CDEBUG(D_QUOTA, ...)</tt>" and the user will get an error at their application.</p> <p>As a general cleanup, it would also be better to move the <tt>QUOTA_FL_&#42;</tt> flags over to a named enum like "<tt>enum quota_local_flags</tt>" or similar so that it is easier to see what this field actually holds, rather than a totally generic "<tt>int *flags</tt>" argument that has to be hunted through the code. </p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36906" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36906</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13043" title="Project quotas: force to ignore quota flags =8 on quota exceeded" class="issue-link" data-issue-key="LU-13043"><del>LU-13043</del></a> quota: remove annoying message in osd_declare_inode_qid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0e381329307852f058f31463cd33f183ff205afa</p> <p>Thanks for the quick patch! I just had an opportunity to deploy it on one OSS and I confirm no message can be seen anymore when a user reaches the project's block quota limit.</p> <p>We have deployed the patch on all servers.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36906/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36906/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13043" title="Project quotas: force to ignore quota flags =8 on quota exceeded" class="issue-link" data-issue-key="LU-13043"><del>LU-13043</del></a> quota: remove annoying message in osd_declare_inode_qid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b3005155317b27e19c8029e6a9f92e69d0dd905e</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37131" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37131</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13043" title="Project quotas: force to ignore quota flags =8 on quota exceeded" class="issue-link" data-issue-key="LU-13043"><del>LU-13043</del></a> quota: remove annoying message in osd_declare_inode_qid()<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a5e07514ef004c9a5a61b62d9942b6eac5b6aafc</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37131/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37131/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13043" title="Project quotas: force to ignore quota flags =8 on quota exceeded" class="issue-link" data-issue-key="LU-13043"><del>LU-13043</del></a> quota: remove annoying message in osd_declare_inode_qid()<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 91bc6ffbfd0aa2d54d024fe7f9e5a5b812bb3002</p> Related Development Rank 1|i00q9j: Rank (Obsolete) 9223372036854775807 Severity