[LU-13043] Project quotas: force to ignore quota flags =8 on quota exceeded - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: Lustre 2.14.0, Lustre 2.12.4
Affects Version/s: Lustre 2.12.3
Labels:
None
Environment:
CentOS 7.6 / 2.12.3 servers and clients

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

We're performing the last tests before putting Project quotas in production on Sherlock/Fir. We noticed today that when using enforced project quotas, with a default quota limits set, we can see a lot of error messages on all OSS when a user reaches the block quota limit (not the inode limit):

fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50958:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 112944:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 50909:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io7-s1: Dec 02 15:34:47 fir-io7-s1 kernel: LustreError: 51093:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43806:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 43917:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 105082:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 55797:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:20 fir-io8-s1 kernel: LustreError: 62343:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 84857:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 120934:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58191:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:26 fir-io5-s2 kernel: LustreError: 58092:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 109202:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io6-s1: Dec 02 15:35:31 fir-io6-s1 kernel: LustreError: 48652:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33426:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33422:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33427:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74530:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74518:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74519:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 33395:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:35:36 fir-io4-s2 kernel: LustreError: 74482:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s2: Dec 02 15:35:44 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 66898:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s2: Dec 02 15:35:45 fir-io8-s2 kernel: LustreError: 3759:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s1: Dec 02 15:35:53 fir-io8-s1 kernel: LustreError: 105052:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io5-s2: Dec 02 15:35:58 fir-io5-s2 kernel: LustreError: 58172:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io6-s1: Dec 02 15:36:03 fir-io6-s1 kernel: LustreError: 48651:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io4-s2: Dec 02 15:36:03 fir-io4-s2 kernel: LustreError: 74524:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8
fir-io8-s2: Dec 02 15:36:03 fir-io8-s2 kernel: LustreError: 66965:0:(osd_quota.c:708:osd_declare_inode_qid()) force to ignore quota flags =8

Block and inode quota limits are properly enforced, we're just curious to know how we can disable these messages.

This is the output of lfs quota when reaching the block quota limits and the messages appear on the servers:

[sthiell@sh-ln07 login /scratch/users/sthiell/sink]$ lfs quota -vp $(id -u) /scratch
Disk quotas for prj 282232 (pid 282232):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /scratch 45001977720* 45000000000 45000000000       - 20409287  30000000 30000000       -
fir-MDT0000_UUID
                10257956*      - 10257956       - 20409287       - 20764938       -
fir-MDT0001_UUID
                      0       -       4       -       0       -       1       -
fir-MDT0002_UUID
                      0       - 13753024       -       0       - 4194304       -
fir-MDT0003_UUID
                      0       -       4       -       0       -       1       -
fir-OST0000_UUID
                464361544*      - 464361544       -       -       -       -       -
fir-OST0001_UUID
                559676264*      - 559676264       -       -       -       -       -
fir-OST0002_UUID
                507091492       - 507092496       -       -       -       -       -
fir-OST0003_UUID
                392618344*      - 392618344       -       -       -       -       -
fir-OST0004_UUID
                942404472       - 942404480       -       -       -       -       -
fir-OST0005_UUID
                449001624*      - 449001624       -       -       -       -       -
fir-OST0006_UUID
                397667212*      - 397667212       -       -       -       -       -
fir-OST0007_UUID
                490947300*      - 490947300       -       -       -       -       -
fir-OST0008_UUID
                465962552       - 465962592       -       -       -       -       -
fir-OST0009_UUID
                391662704*      - 391662704       -       -       -       -       -
fir-OST000a_UUID
                448462700*      - 448462700       -       -       -       -       -
fir-OST000b_UUID
                285858668*      - 285858668       -       -       -       -       -
fir-OST000c_UUID
                452055424*      - 452055424       -       -       -       -       -
fir-OST000d_UUID
                664806004*      - 664806004       -       -       -       -       -
fir-OST000e_UUID
                435402460*      - 435402460       -       -       -       -       -
fir-OST000f_UUID
                357099424*      - 357099424       -       -       -       -       -
fir-OST0010_UUID
                571025464*      - 571025464       -       -       -       -       -
fir-OST0011_UUID
                476045756*      - 476045756       -       -       -       -       -
fir-OST0012_UUID
                401734136*      - 401734136       -       -       -       -       -
fir-OST0013_UUID
                563884544*      - 563884544       -       -       -       -       -
fir-OST0014_UUID
                451027312       - 451028316       -       -       -       -       -
fir-OST0015_UUID
                1008410220*      - 1008410220       -       -       -       -       -
fir-OST0016_UUID
                459554296*      - 459554296       -       -       -       -       -
fir-OST0017_UUID
                406426536       - 406427548       -       -       -       -       -
fir-OST0018_UUID
                431504956       - 431505964       -       -       -       -       -
fir-OST0019_UUID
                488622276*      - 488622276       -       -       -       -       -
fir-OST001a_UUID
                374191672*      - 374191672       -       -       -       -       -
fir-OST001b_UUID
                468069876       - 468069908       -       -       -       -       -
fir-OST001c_UUID
                488161748*      - 488161748       -       -       -       -       -
fir-OST001d_UUID
                500656704*      - 500656704       -       -       -       -       -
fir-OST001e_UUID
                552511176*      - 552511176       -       -       -       -       -
fir-OST001f_UUID
                500188684*      - 500188684       -       -       -       -       -
fir-OST0020_UUID
                356752432*      - 356752432       -       -       -       -       -
fir-OST0021_UUID
                514022332*      - 514022332       -       -       -       -       -
fir-OST0022_UUID
                388244648       - 388245656       -       -       -       -       -
fir-OST0023_UUID
                478265396*      - 478265396       -       -       -       -       -
fir-OST0024_UUID
                662203064       - 662204080       -       -       -       -       -
fir-OST0025_UUID
                357218288       - 357219296       -       -       -       -       -
fir-OST0026_UUID
                388994408*      - 388994408       -       -       -       -       -
fir-OST0027_UUID
                416523512*      - 416523512       -       -       -       -       -
fir-OST0028_UUID
                308196948*      - 308196948       -       -       -       -       -
fir-OST0029_UUID
                387409384*      - 387409384       -       -       -       -       -
fir-OST002a_UUID
                449330736*      - 449330736       -       -       -       -       -
fir-OST002b_UUID
                429416836*      - 429416836       -       -       -       -       -
fir-OST002c_UUID
                394779272*      - 394779272       -       -       -       -       -
fir-OST002d_UUID
                452150536*      - 452150536       -       -       -       -       -
fir-OST002e_UUID
                649498168       - 649499192       -       -       -       -       -
fir-OST002f_UUID
                607361816*      - 607361816       -       -       -       -       -
fir-OST0030_UUID
                214663440*      - 214663440       -       -       -       -       -
fir-OST0031_UUID
                420568284*      - 420568284       -       -       -       -       -
fir-OST0032_UUID
                380047312       - 380048316       -       -       -       -       -
fir-OST0033_UUID
                1057671372*      - 1057671372       -       -       -       -       -
fir-OST0034_UUID
                996787540       - 996788524       -       -       -       -       -
fir-OST0035_UUID
                1088752228*      - 1088752228       -       -       -       -       -
fir-OST0036_UUID
                481519020*      - 481519020       -       -       -       -       -
fir-OST0037_UUID
                323877700*      - 323877700       -       -       -       -       -
fir-OST0038_UUID
                272179672*      - 272179672       -       -       -       -       -
fir-OST0039_UUID
                475962020*      - 475962020       -       -       -       -       -
fir-OST003a_UUID
                291771688*      - 291771688       -       -       -       -       -
fir-OST003b_UUID
                248632172*      - 248632172       -       -       -       -       -
fir-OST003c_UUID
                632507520*      - 632507520       -       -       -       -       -
fir-OST003d_UUID
                383028632*      - 383028632       -       -       -       -       -
fir-OST003e_UUID
                355118096*      - 355118096       -       -       -       -       -
fir-OST003f_UUID
                787302308*      - 787302308       -       -       -       -       -
fir-OST0040_UUID
                330605936       - 330606928       -       -       -       -       -
fir-OST0041_UUID
                453859240*      - 453859240       -       -       -       -       -
fir-OST0042_UUID
                440181540*      - 440181540       -       -       -       -       -
fir-OST0043_UUID
                261950232*      - 261950232       -       -       -       -       -
fir-OST0044_UUID
                216148944*      - 216148944       -       -       -       -       -
fir-OST0045_UUID
                416328968*      - 416328968       -       -       -       -       -
fir-OST0046_UUID
                173872668*      - 173872668       -       -       -       -       -
fir-OST0047_UUID
                852739020       - 852740028       -       -       -       -       -
fir-OST0048_UUID
                377475652*      - 377475652       -       -       -       -       -
fir-OST0049_UUID
                265111876       - 265111908       -       -       -       -       -
fir-OST004a_UUID
                400281672*      - 400281672       -       -       -       -       -
fir-OST004b_UUID
                414808608*      - 414808608       -       -       -       -       -
fir-OST004c_UUID
                138487876*      - 138487876       -       -       -       -       -
fir-OST004d_UUID
                564593396*      - 564593396       -       -       -       -       -
fir-OST004e_UUID
                218610204*      - 218610204       -       -       -       -       -
fir-OST004f_UUID
                413015672*      - 413015672       -       -       -       -       -
fir-OST0050_UUID
                508523192       - 508524216       -       -       -       -       -
fir-OST0051_UUID
                581310148*      - 581310148       -       -       -       -       -
fir-OST0052_UUID
                198038980*      - 198038980       -       -       -       -       -
fir-OST0053_UUID
                360050332*      - 360050332       -       -       -       -       -
fir-OST0054_UUID
                229448744*      - 229448744       -       -       -       -       -
fir-OST0055_UUID
                559147340*      - 559147340       -       -       -       -       -
fir-OST0056_UUID
                532999060*      - 532999060       -       -       -       -       -
fir-OST0057_UUID
                339694332       - 339695352       -       -       -       -       -
fir-OST0058_UUID
                986951040*      - 986951040       -       -       -       -       -
fir-OST0059_UUID
                175977116*      - 175977116       -       -       -       -       -
fir-OST005a_UUID
                372989596*      - 372989596       -       -       -       -       -
fir-OST005b_UUID
                662420908*      - 662420908       -       -       -       -       -
fir-OST005c_UUID
                445986612*      - 445986612       -       -       -       -       -
fir-OST005d_UUID
                302969960*      - 302969960       -       -       -       -       -
fir-OST005e_UUID
                453619292*      - 453619292       -       -       -       -       -
fir-OST005f_UUID
                845671284*      - 845671284       -       -       -       -       -
Total allocated inode limit: 24959244, total allocated block limit: 44991733992
pid 282232 is using default block quota setting
pid 282232 is using default file quota setting

Attachments

Activity

[LU-13043] Project quotas: force to ignore quota flags =8 on quota exceeded

Gerrit Updater added a comment - 10/Jan/20 8:05 AM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37131/
Subject: ~~LU-13043~~ quota: remove annoying message in osd_declare_inode_qid()
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 91bc6ffbfd0aa2d54d024fe7f9e5a5b812bb3002

Gerrit Updater added a comment - 10/Jan/20 8:05 AM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37131/ Subject: LU-13043 quota: remove annoying message in osd_declare_inode_qid() Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 91bc6ffbfd0aa2d54d024fe7f9e5a5b812bb3002

Gerrit Updater added a comment - 03/Jan/20 3:04 AM

Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37131
Subject: ~~LU-13043~~ quota: remove annoying message in osd_declare_inode_qid()
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: a5e07514ef004c9a5a61b62d9942b6eac5b6aafc

Gerrit Updater added a comment - 03/Jan/20 3:04 AM Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37131 Subject: LU-13043 quota: remove annoying message in osd_declare_inode_qid() Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: a5e07514ef004c9a5a61b62d9942b6eac5b6aafc

Peter Jones added a comment - 03/Jan/20 2:41 AM

Landed for 2.14

Peter Jones added a comment - 03/Jan/20 2:41 AM Landed for 2.14

Gerrit Updater added a comment - 03/Jan/20 12:09 AM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36906/
Subject: ~~LU-13043~~ quota: remove annoying message in osd_declare_inode_qid()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b3005155317b27e19c8029e6a9f92e69d0dd905e

Gerrit Updater added a comment - 03/Jan/20 12:09 AM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36906/ Subject: LU-13043 quota: remove annoying message in osd_declare_inode_qid() Project: fs/lustre-release Branch: master Current Patch Set: Commit: b3005155317b27e19c8029e6a9f92e69d0dd905e

Stephane Thiell added a comment - 10/Dec/19 5:57 PM

We have deployed the patch on all servers.

Stephane Thiell added a comment - 10/Dec/19 5:57 PM We have deployed the patch on all servers.

Stephane Thiell added a comment - 06/Dec/19 7:38 PM

Thanks for the quick patch! I just had an opportunity to deploy it on one OSS and I confirm no message can be seen anymore when a user reaches the project's block quota limit.

Stephane Thiell added a comment - 06/Dec/19 7:38 PM Thanks for the quick patch! I just had an opportunity to deploy it on one OSS and I confirm no message can be seen anymore when a user reaches the project's block quota limit.

Gerrit Updater added a comment - 03/Dec/19 6:47 AM

Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36906
Subject: ~~LU-13043~~ quota: remove annoying message in osd_declare_inode_qid()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0e381329307852f058f31463cd33f183ff205afa

Gerrit Updater added a comment - 03/Dec/19 6:47 AM Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36906 Subject: LU-13043 quota: remove annoying message in osd_declare_inode_qid() Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 0e381329307852f058f31463cd33f183ff205afa

Andreas Dilger added a comment - 03/Dec/19 12:10 AM

Looking into this quickly, it appears that "QUOTA_FL_OVER_PRJQUOTA = 8" but this does not have a matching OSD_QID_PRJ definition in enum osd_qid_declare_flags. Using "CDEBUG(D_ERROR, ...)" for this message is bad, since this prints to the console and does not rate limit the messages. At a very minimum, this should be changed to "CERROR()" so the message is rate limited, but I question whether this message is needed on the console at all. The admin shouldn't be getting console error messages when a user goes over quota (this would be happening continuously at some sites), so this would probably be better as "CDEBUG(D_QUOTA, ...)" and the user will get an error at their application.

As a general cleanup, it would also be better to move the QUOTA_FL_* flags over to a named enum like "enum quota_local_flags" or similar so that it is easier to see what this field actually holds, rather than a totally generic "int *flags" argument that has to be hunted through the code.

Andreas Dilger added a comment - 03/Dec/19 12:10 AM Looking into this quickly, it appears that " QUOTA_FL_OVER_PRJQUOTA = 8 " but this does not have a matching OSD_QID_PRJ definition in enum osd_qid_declare_flags . Using " CDEBUG(D_ERROR, ...) " for this message is bad, since this prints to the console and does not rate limit the messages. At a very minimum, this should be changed to " CERROR() " so the message is rate limited, but I question whether this message is needed on the console at all. The admin shouldn't be getting console error messages when a user goes over quota (this would be happening continuously at some sites), so this would probably be better as " CDEBUG(D_QUOTA, ...) " and the user will get an error at their application. As a general cleanup, it would also be better to move the QUOTA_FL_* flags over to a named enum like " enum quota_local_flags " or similar so that it is easier to see what this field actually holds, rather than a totally generic " int *flags " argument that has to be hunted through the code.

Peter Jones added a comment - 02/Dec/19 11:56 PM

Shilong

Can you please advise?

Thanks

Peter

Peter Jones added a comment - 02/Dec/19 11:56 PM Shilong Can you please advise? Thanks Peter

People

Assignee:: Wang Shilong (Inactive)

Reporter:: Stephane Thiell

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 02/Dec/19 11:52 PM

Updated:: 10/Jan/20 12:56 PM

Resolved:: 03/Jan/20 2:41 AM