Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12557

Wrong MDT inode quota limits lead to quota not enforced

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • None
    • Lustre 2.10.8
    • None
    • CentOS 7 - Lustre 2.10.8 clients and servers
    • 3
    • 9223372036854775807

    Description

      This morning, we found a group on Oak that was able to get past their inode quota. The limit should be 1.5M inodes, but they were able to get past it at about 1.9M inodes. We have checked that this didn't involve Lustre 2.12 (tested on a 2.10.8 client also just to be sure it was not a 2.10/2.12 compat issue).

       

      [root@oak-gw02 ~]# lfs quota -g oak_hlwill /oak
      Disk quotas for grp oak_hlwill (gid 6324):
           Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
                 /oak 8533764076  10000000000 10000000000       - 1899647* 1500000 1500000       -
      [root@oak-gw02 ~]# lfs quota -v -g oak_hlwill /oak
      Disk quotas for grp oak_hlwill (gid 6324):
           Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
                 /oak 8533764076  10000000000 10000000000       - 1899647* 1500000 1500000       -
      oak-MDT0000_UUID
                            0       -       0       -       0       - 1227848       -
      oak-MDT0001_UUID
                       191560       -       0       - 1899647       - 1900270       -
      oak-OST0000_UUID
                      17351952       - 31759736       -       -       -       -       -
      oak-OST0001_UUID
                      66026588       - 81822144       -       -       -       -       -
      oak-OST0002_UUID
                      14922776       - 31511992       -       -       -       -       -
      oak-OST0003_UUID
                      15177156       - 31764608       -       -       -       -       -
      oak-OST0004_UUID
                      24677808       - 40406296       -       -       -       -       -
      oak-OST0005_UUID
                      66847632       - 77731728       -       -       -       -       -
      oak-OST0006_UUID
                      41358308       - 47813388       -       -       -       -       -
      oak-OST0007_UUID
                      51010844       - 62061088       -       -       -       -       -
      oak-OST0008_UUID
                      27771804       - 44114220       -       -       -       -       -
      oak-OST0009_UUID
                      14601076       - 19432308       -       -       -       -       -
      oak-OST000a_UUID
                      34389472       - 47193616       -       -       -       -       -
      oak-OST000b_UUID
                      37729512       - 47398016       -       -       -       -       -
      oak-OST000c_UUID
                      19129260       - 31648056       -       -       -       -       -
      oak-OST000d_UUID
                      86069244       - 94557420       -       -       -       -       -
      oak-OST000e_UUID
                      102349016       - 110564620       -       -       -       -       -
      oak-OST000f_UUID
                      54054828       - 63654276       -       -       -       -       -
      oak-OST0010_UUID
                      65474316       - 78720760       -       -       -       -       -
      oak-OST0011_UUID
                      35672936       - 47747340       -       -       -       -       -
      oak-OST0012_UUID
                      34334216       - 48068880       -       -       -       -       -
      oak-OST0013_UUID
                      174346708       - 189803564       -       -       -       -       -
      oak-OST0014_UUID
                      35482060       - 48985040       -       -       -       -       -
      oak-OST0015_UUID
                      30785012       - 47307496       -       -       -       -       -
      oak-OST0016_UUID
                      14288204       - 30010196       -       -       -       -       -
      oak-OST0017_UUID
                      160723328       - 170963160       -       -       -       -       -
      oak-OST0018_UUID
                      29121536       - 45781220       -       -       -       -       -
      oak-OST0019_UUID
                      36321604       - 50362328       -       -       -       -       -
      oak-OST001a_UUID
                      20412724       - 31637720       -       -       -       -       -
      oak-OST001b_UUID
                      182895796       - 196926944       -       -       -       -       -
      oak-OST001c_UUID
                      19553908       - 32030544       -       -       -       -       -
      oak-OST001d_UUID
                      62042768       - 78075652       -       -       -       -       -
      oak-OST001e_UUID
                      44483932       - 61192240       -       -       -       -       -
      oak-OST001f_UUID
                      56387556       - 62735272       -       -       -       -       -
      oak-OST0020_UUID
                      48325020       - 52707804       -       -       -       -       -
      oak-OST0021_UUID
                      21045360       - 32513732       -       -       -       -       -
      oak-OST0022_UUID
                      24929796       - 32562208       -       -       -       -       -
      oak-OST0023_UUID
                      16102080       - 32269368       -       -       -       -       -
      oak-OST0024_UUID
                      33986196       - 48985176       -       -       -       -       -
      oak-OST0025_UUID
                      13570580       - 29364152       -       -       -       -       -
      oak-OST0026_UUID
                      45386672       - 62043520       -       -       -       -       -
      oak-OST0027_UUID
                      13126336       - 29150520       -       -       -       -       -
      oak-OST0028_UUID
                      20959648       - 32092356       -       -       -       -       -
      oak-OST0029_UUID
                      53692068       - 63209868       -       -       -       -       -
      oak-OST002a_UUID
                      67681888       - 77166948       -       -       -       -       -
      oak-OST002b_UUID
                      24941128       - 31475784       -       -       -       -       -
      oak-OST002c_UUID
                      22913216       - 32134972       -       -       -       -       -
      oak-OST002d_UUID
                      15253452       - 31461332       -       -       -       -       -
      oak-OST002e_UUID
                      31644152       - 47380988       -       -       -       -       -
      oak-OST002f_UUID
                      77376284       - 93805364       -       -       -       -       -
      oak-OST0030_UUID
                      33129164       - 46462924       -       -       -       -       -
      oak-OST0031_UUID
                      32701444       - 47223908       -       -       -       -       -
      oak-OST0032_UUID
                      55045664       - 63461196       -       -       -       -       -
      oak-OST0033_UUID
                      19591076       - 30847104       -       -       -       -       -
      oak-OST0034_UUID
                      25803956       - 32684668       -       -       -       -       -
      oak-OST0035_UUID
                      17417308       - 32159544       -       -       -       -       -
      oak-OST0036_UUID
                      34644580       - 42654224       -       -       -       -       -
      oak-OST0037_UUID
                      55375144       - 63712260       -       -       -       -       -
      oak-OST0038_UUID
                      17038824       - 31374628       -       -       -       -       -
      oak-OST0039_UUID
                      39546648       - 47353392       -       -       -       -       -
      oak-OST003a_UUID
                      102553208       - 110268964       -       -       -       -       -
      oak-OST003b_UUID
                      28840752       - 44928480       -       -       -       -       -
      oak-OST003c_UUID
                      49202920       - 63133228       -       -       -       -       -
      oak-OST003d_UUID
                      49559824       - 63497608       -       -       -       -       -
      oak-OST003e_UUID
                      13722604       - 30024312       -       -       -       -       -
      oak-OST003f_UUID
                      70378068       - 78122168       -       -       -       -       -
      oak-OST0040_UUID
                      68462244       - 77360832       -       -       -       -       -
      oak-OST0041_UUID
                      57653700       - 63235376       -       -       -       -       -
      oak-OST0042_UUID
                      20172548       - 32054296       -       -       -       -       -
      oak-OST0043_UUID
                      37357332       - 48946684       -       -       -       -       -
      oak-OST0044_UUID
                      126518512       - 141769980       -       -       -       -       -
      oak-OST0045_UUID
                      67839960       - 78785684       -       -       -       -       -
      oak-OST0046_UUID
                      39016316       - 55793244       -       -       -       -       -
      oak-OST0047_UUID
                      17682040       - 31681328       -       -       -       -       -
      oak-OST0048_UUID
                      240605844       - 251938788       -       -       -       -       -
      oak-OST0049_UUID
                      30177248       - 46168020       -       -       -       -       -
      oak-OST004a_UUID
                      44678516       - 58587588       -       -       -       -       -
      oak-OST004b_UUID
                      61939084       - 78400116       -       -       -       -       -
      oak-OST004c_UUID
                      21858052       - 34955536       -       -       -       -       -
      oak-OST004d_UUID
                      138522544       - 154789120       -       -       -       -       -
      oak-OST004e_UUID
                      49504748       - 63841892       -       -       -       -       -
      oak-OST004f_UUID
                      130926420       - 146466620       -       -       -       -       -
      oak-OST0050_UUID
                      57729864       - 63365516       -       -       -       -       -
      oak-OST0051_UUID
                      48460656       - 63959460       -       -       -       -       -
      oak-OST0052_UUID
                      94210388       - 109956288       -       -       -       -       -
      oak-OST0053_UUID
                      64437928       - 78429300       -       -       -       -       -
      oak-OST0054_UUID
                      42852680       - 53818780       -       -       -       -       -
      oak-OST0055_UUID
                      183838744       - 199938748       -       -       -       -       -
      oak-OST0056_UUID
                      39198436       - 46140168       -       -       -       -       -
      oak-OST0057_UUID
                      68744424       - 84933852       -       -       -       -       -
      oak-OST0058_UUID
                      31213976       - 47337560       -       -       -       -       -
      oak-OST0059_UUID
                      57913780       - 74479388       -       -       -       -       -
      oak-OST005a_UUID
                      82979688       - 97595716       -       -       -       -       -
      oak-OST005b_UUID
                      67481288       - 79103856       -       -       -       -       -
      oak-OST005c_UUID
                      97845408       - 112712476       -       -       -       -       -
      oak-OST005d_UUID
                      71769900       - 88129176       -       -       -       -       -
      oak-OST005e_UUID
                      40552544       - 46876768       -       -       -       -       -
      oak-OST005f_UUID
                      30870984       - 46878048       -       -       -       -       -
      oak-OST0060_UUID
                      54666344       -       0       -       -       -       -       -
      oak-OST0061_UUID
                      57431844       -       0       -       -       -       -       -
      oak-OST0062_UUID
                      137714524       -       0       -       -       -       -       -
      oak-OST0063_UUID
                      22874128       -       0       -       -       -       -       -
      oak-OST0064_UUID
                      76953932       -       0       -       -       -       -       -
      oak-OST0065_UUID
                      103531832       -       0       -       -       -       -       -
      oak-OST0066_UUID
                      71837956       -       0       -       -       -       -       -
      oak-OST0067_UUID
                      46396392       -       0       -       -       -       -       -
      oak-OST0068_UUID
                      132340672       -       0       -       -       -       -       -
      oak-OST0069_UUID
                      39410204       -       0       -       -       -       -       -
      oak-OST006a_UUID
                      43926100       -       0       -       -       -       -       -
      oak-OST006b_UUID
                      154165452       -       0       -       -       -       -       -
      oak-OST006c_UUID
                      210254328       -       0       -       -       -       -       -
      oak-OST006d_UUID
                      35547240       -       0       -       -       -       -       -
      oak-OST006e_UUID
                      49122000       -       0       -       -       -       -       -
      oak-OST006f_UUID
                      44907184       -       0       -       -       -       -       -
      oak-OST0070_UUID
                      43110756       -       0       -       -       -       -       -
      oak-OST0071_UUID
                      50232656       -       0       -       -       -       -       -
      oak-OST0072_UUID
                      244608820       -       0       -       -       -       -       -
      oak-OST0073_UUID
                      90048308       -       0       -       -       -       -       -
      oak-OST0074_UUID
                      78561112       -       0       -       -       -       -       -
      oak-OST0075_UUID
                      81974144       -       0       -       -       -       -       -
      oak-OST0076_UUID
                      235628012       -       0       -       -       -       -       -
      oak-OST0077_UUID
                      279483236       -       0       -       -       -       -       -
      oak-OST0078_UUID
                      139394324       -       0       -       -       -       -       -
      oak-OST0079_UUID
                      122266916       -       0       -       -       -       -       -
      oak-OST007a_UUID
                      194638420       -       0       -       -       -       -       -
      oak-OST007b_UUID
                      131759012       -       0       -       -       -       -       -
      oak-OST007c_UUID
                      342335656       -       0       -       -       -       -       -
      oak-OST007d_UUID
                      130494584       -       0       -       -       -       -       -
      oak-OST007e_UUID
                         1632       -       0       -       -       -       -       -
      oak-OST007f_UUID
                       125044       -       0       -       -       -       -       -
      oak-OST0080_UUID
                        52284       -       0       -       -       -       -       -
      oak-OST0081_UUID
                        70460       -       0       -       -       -       -       -
      oak-OST0082_UUID
                        15648       -       0       -       -       -       -       -
      oak-OST0083_UUID
                       510460       -       0       -       -       -       -       -
      oak-OST0084_UUID
                       138332       -       0       -       -       -       -       -
      oak-OST0085_UUID
                        84700       -       0       -       -       -       -       -
      oak-OST0086_UUID
                      1024384       -       0       -       -       -       -       -
      oak-OST0087_UUID
                        76132       -       0       -       -       -       -       -
      oak-OST0088_UUID
                        26312       -       0       -       -       -       -       -
      oak-OST0089_UUID
                        68328       -       0       -       -       -       -       -
      oak-OST008a_UUID
                        59324       -       0       -       -       -       -       -
      oak-OST008b_UUID
                       268480       -       0       -       -       -       -       -
      oak-OST008c_UUID
                        41144       -       0       -       -       -       -       -
      oak-OST008d_UUID
                       267472       -       0       -       -       -       -       -
      oak-OST008e_UUID
                       120212       -       0       -       -       -       -       -
      oak-OST008f_UUID
                       711344       -       0       -       -       -       -       -
      Total allocated inode limit: 3128118, total allocated block limit: 6298444772
      
      # clush -bw@mds 'lctl get_param osd-*.*.quota_slave.info'
      ---------------
      oak-md1-s1
      ---------------
      osd-ldiskfs.oak-MDT0001.quota_slave.info=
      target name:    oak-MDT0001
      pool ID:        0
      type:           md
      quota enabled:  g
      conn to master: setup
      space acct:     ug
      user uptodate:  glb[0],slv[0],reint[0]
      group uptodate: glb[1],slv[1],reint[0]
      project uptodate: glb[0],slv[0],reint[0]
      ---------------
      oak-md1-s2
      ---------------
      osd-ldiskfs.oak-MDT0000.quota_slave.info=
      target name:    oak-MDT0000
      pool ID:        0
      type:           md
      quota enabled:  g
      conn to master: setup
      space acct:     ug
      user uptodate:  glb[0],slv[0],reint[0]
      group uptodate: glb[1],slv[1],reint[0]
      project uptodate: glb[0],slv[0],reint[0]
      

      Group quota is not enforced at this time, I was able to create new files in this group. Any suggestion?
       

      Attachments

        Activity

          People

            hongchao.zhang Hongchao Zhang
            sthiell Stephane Thiell
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: