[LU-13043] Project quotas: force to ignore quota flags =8 on quota exceeded Created: 02/Dec/19 Updated: 10/Jan/20 Resolved: 03/Jan/20 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.12.3 |
| Fix Version/s: | Lustre 2.14.0, Lustre 2.12.4 |
| Type: | Bug | Priority: | Minor |
| Reporter: | Stephane Thiell | Assignee: | Wang Shilong (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Environment: |
CentOS 7.6 / 2.12.3 servers and clients |
||
| Issue Links: |
|
||||
| Severity: | 3 | ||||
| Rank (Obsolete): | 9223372036854775807 | ||||
| Description |
|
We're performing the last tests before putting Project quotas in production on Sherlock/Fir. We noticed today that when using enforced project quotas, with a default quota limits set, we can see a lot of error messages on all OSS when a user reaches the block quota limit (not the inode limit):
fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50958:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 112944:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50909:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 51093:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43806:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43917:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 105082:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 55797:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 62343:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 84857:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 120934:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58191:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58092:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 109202:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 48652:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33426:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33422:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33427:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74530:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74518:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74519:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33395:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74482:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:44 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 66898:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 3759:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s1: Dec 02 15:35:53 fir-io8-s1 kernel: LustreError: 105052:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io5-s2: Dec 02 15:35:58 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io6-s1: Dec 02 15:36:03 fir-io6-s1 kernel: LustreError: 48651:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io4-s2: Dec 02 15:36:03 fir-io4-s2 kernel: LustreError: 74524:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 fir-io8-s2: Dec 02 15:36:03 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8 Block and inode quota limits are properly enforced, we're just curious to know how we can disable these messages. This is the output of lfs quota when reaching the block quota limits and the messages appear on the servers: [sthiell@sh-ln07 login /scratch/users/sthiell/sink]$ lfs quota -vp $(id -u) /scratch
Disk quotas for prj 282232 (pid 282232):
Filesystem kbytes quota limit grace files quota limit grace
/scratch 45001977720* 45000000000 45000000000 - 20409287 30000000 30000000 -
fir-MDT0000_UUID
10257956* - 10257956 - 20409287 - 20764938 -
fir-MDT0001_UUID
0 - 4 - 0 - 1 -
fir-MDT0002_UUID
0 - 13753024 - 0 - 4194304 -
fir-MDT0003_UUID
0 - 4 - 0 - 1 -
fir-OST0000_UUID
464361544* - 464361544 - - - - -
fir-OST0001_UUID
559676264* - 559676264 - - - - -
fir-OST0002_UUID
507091492 - 507092496 - - - - -
fir-OST0003_UUID
392618344* - 392618344 - - - - -
fir-OST0004_UUID
942404472 - 942404480 - - - - -
fir-OST0005_UUID
449001624* - 449001624 - - - - -
fir-OST0006_UUID
397667212* - 397667212 - - - - -
fir-OST0007_UUID
490947300* - 490947300 - - - - -
fir-OST0008_UUID
465962552 - 465962592 - - - - -
fir-OST0009_UUID
391662704* - 391662704 - - - - -
fir-OST000a_UUID
448462700* - 448462700 - - - - -
fir-OST000b_UUID
285858668* - 285858668 - - - - -
fir-OST000c_UUID
452055424* - 452055424 - - - - -
fir-OST000d_UUID
664806004* - 664806004 - - - - -
fir-OST000e_UUID
435402460* - 435402460 - - - - -
fir-OST000f_UUID
357099424* - 357099424 - - - - -
fir-OST0010_UUID
571025464* - 571025464 - - - - -
fir-OST0011_UUID
476045756* - 476045756 - - - - -
fir-OST0012_UUID
401734136* - 401734136 - - - - -
fir-OST0013_UUID
563884544* - 563884544 - - - - -
fir-OST0014_UUID
451027312 - 451028316 - - - - -
fir-OST0015_UUID
1008410220* - 1008410220 - - - - -
fir-OST0016_UUID
459554296* - 459554296 - - - - -
fir-OST0017_UUID
406426536 - 406427548 - - - - -
fir-OST0018_UUID
431504956 - 431505964 - - - - -
fir-OST0019_UUID
488622276* - 488622276 - - - - -
fir-OST001a_UUID
374191672* - 374191672 - - - - -
fir-OST001b_UUID
468069876 - 468069908 - - - - -
fir-OST001c_UUID
488161748* - 488161748 - - - - -
fir-OST001d_UUID
500656704* - 500656704 - - - - -
fir-OST001e_UUID
552511176* - 552511176 - - - - -
fir-OST001f_UUID
500188684* - 500188684 - - - - -
fir-OST0020_UUID
356752432* - 356752432 - - - - -
fir-OST0021_UUID
514022332* - 514022332 - - - - -
fir-OST0022_UUID
388244648 - 388245656 - - - - -
fir-OST0023_UUID
478265396* - 478265396 - - - - -
fir-OST0024_UUID
662203064 - 662204080 - - - - -
fir-OST0025_UUID
357218288 - 357219296 - - - - -
fir-OST0026_UUID
388994408* - 388994408 - - - - -
fir-OST0027_UUID
416523512* - 416523512 - - - - -
fir-OST0028_UUID
308196948* - 308196948 - - - - -
fir-OST0029_UUID
387409384* - 387409384 - - - - -
fir-OST002a_UUID
449330736* - 449330736 - - - - -
fir-OST002b_UUID
429416836* - 429416836 - - - - -
fir-OST002c_UUID
394779272* - 394779272 - - - - -
fir-OST002d_UUID
452150536* - 452150536 - - - - -
fir-OST002e_UUID
649498168 - 649499192 - - - - -
fir-OST002f_UUID
607361816* - 607361816 - - - - -
fir-OST0030_UUID
214663440* - 214663440 - - - - -
fir-OST0031_UUID
420568284* - 420568284 - - - - -
fir-OST0032_UUID
380047312 - 380048316 - - - - -
fir-OST0033_UUID
1057671372* - 1057671372 - - - - -
fir-OST0034_UUID
996787540 - 996788524 - - - - -
fir-OST0035_UUID
1088752228* - 1088752228 - - - - -
fir-OST0036_UUID
481519020* - 481519020 - - - - -
fir-OST0037_UUID
323877700* - 323877700 - - - - -
fir-OST0038_UUID
272179672* - 272179672 - - - - -
fir-OST0039_UUID
475962020* - 475962020 - - - - -
fir-OST003a_UUID
291771688* - 291771688 - - - - -
fir-OST003b_UUID
248632172* - 248632172 - - - - -
fir-OST003c_UUID
632507520* - 632507520 - - - - -
fir-OST003d_UUID
383028632* - 383028632 - - - - -
fir-OST003e_UUID
355118096* - 355118096 - - - - -
fir-OST003f_UUID
787302308* - 787302308 - - - - -
fir-OST0040_UUID
330605936 - 330606928 - - - - -
fir-OST0041_UUID
453859240* - 453859240 - - - - -
fir-OST0042_UUID
440181540* - 440181540 - - - - -
fir-OST0043_UUID
261950232* - 261950232 - - - - -
fir-OST0044_UUID
216148944* - 216148944 - - - - -
fir-OST0045_UUID
416328968* - 416328968 - - - - -
fir-OST0046_UUID
173872668* - 173872668 - - - - -
fir-OST0047_UUID
852739020 - 852740028 - - - - -
fir-OST0048_UUID
377475652* - 377475652 - - - - -
fir-OST0049_UUID
265111876 - 265111908 - - - - -
fir-OST004a_UUID
400281672* - 400281672 - - - - -
fir-OST004b_UUID
414808608* - 414808608 - - - - -
fir-OST004c_UUID
138487876* - 138487876 - - - - -
fir-OST004d_UUID
564593396* - 564593396 - - - - -
fir-OST004e_UUID
218610204* - 218610204 - - - - -
fir-OST004f_UUID
413015672* - 413015672 - - - - -
fir-OST0050_UUID
508523192 - 508524216 - - - - -
fir-OST0051_UUID
581310148* - 581310148 - - - - -
fir-OST0052_UUID
198038980* - 198038980 - - - - -
fir-OST0053_UUID
360050332* - 360050332 - - - - -
fir-OST0054_UUID
229448744* - 229448744 - - - - -
fir-OST0055_UUID
559147340* - 559147340 - - - - -
fir-OST0056_UUID
532999060* - 532999060 - - - - -
fir-OST0057_UUID
339694332 - 339695352 - - - - -
fir-OST0058_UUID
986951040* - 986951040 - - - - -
fir-OST0059_UUID
175977116* - 175977116 - - - - -
fir-OST005a_UUID
372989596* - 372989596 - - - - -
fir-OST005b_UUID
662420908* - 662420908 - - - - -
fir-OST005c_UUID
445986612* - 445986612 - - - - -
fir-OST005d_UUID
302969960* - 302969960 - - - - -
fir-OST005e_UUID
453619292* - 453619292 - - - - -
fir-OST005f_UUID
845671284* - 845671284 - - - - -
Total allocated inode limit: 24959244, total allocated block limit: 44991733992
pid 282232 is using default block quota setting
pid 282232 is using default file quota setting
|
| Comments |
| Comment by Peter Jones [ 02/Dec/19 ] |
|
Shilong Can you please advise? Thanks Peter |
| Comment by Andreas Dilger [ 03/Dec/19 ] |
|
Looking into this quickly, it appears that "QUOTA_FL_OVER_PRJQUOTA = 8" but this does not have a matching OSD_QID_PRJ definition in enum osd_qid_declare_flags. Using "CDEBUG(D_ERROR, ...)" for this message is bad, since this prints to the console and does not rate limit the messages. At a very minimum, this should be changed to "CERROR()" so the message is rate limited, but I question whether this message is needed on the console at all. The admin shouldn't be getting console error messages when a user goes over quota (this would be happening continuously at some sites), so this would probably be better as "CDEBUG(D_QUOTA, ...)" and the user will get an error at their application. As a general cleanup, it would also be better to move the QUOTA_FL_* flags over to a named enum like "enum quota_local_flags" or similar so that it is easier to see what this field actually holds, rather than a totally generic "int *flags" argument that has to be hunted through the code. |
| Comment by Gerrit Updater [ 03/Dec/19 ] |
|
Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36906 |
| Comment by Stephane Thiell [ 06/Dec/19 ] |
|
Thanks for the quick patch! I just had an opportunity to deploy it on one OSS and I confirm no message can be seen anymore when a user reaches the project's block quota limit. |
| Comment by Stephane Thiell [ 10/Dec/19 ] |
|
We have deployed the patch on all servers. |
| Comment by Gerrit Updater [ 03/Jan/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36906/ |
| Comment by Peter Jones [ 03/Jan/20 ] |
|
Landed for 2.14 |
| Comment by Gerrit Updater [ 03/Jan/20 ] |
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37131 |
| Comment by Gerrit Updater [ 10/Jan/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37131/ |