Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-17395

df -h is limited by project quota even if quota enforcement is disabled

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • Lustre 2.16.0
    • None
    • 3
    • 9223372036854775807

    Description

      $ git log --oneline -n3
      d43e8b471f (HEAD -> df_noenf, tag: v2_15_60, tag: 2.15.60, origin/master, origin/HEAD) New tag 2.15.60
      54d8859bef LU-17215 tests: sanity/398q to use $tfile
      7a56a689d4 LU-17374 gss: get rid of rsi cache entries after req handle
      $ MDSCOUNT=2 FSTYPE=ldiskfs bash lustre/tests/llmount.sh
      $ lfs project -sp 1212 /mnt/lustre
      $ lfs setquota -p 1212 -b 10M -B 10M -i 10K -I 10K /mnt/lustre
      $ df -h /mnt/lustre
      Filesystem Size Used Avail Use% Mounted on
      10.0.2.15@tcp:/lustre 10M 4.0K 10M 1% /mnt/lustre
      $ lfs project -d /mnt/lustre
      1212 P /mnt/lustre
      $ lfs project -sp 0 /mnt/lustre
      $ df -h /mnt/lustre
      Filesystem Size Used Avail Use% Mounted on
      10.0.2.15@tcp:/lustre 612M 3.0M 556M 1% /mnt/lustre
      $ lctl get_param osd-ldiskfs.*.quota_slave.info | grep enabled
      quota enabled: none
      quota enabled: none
      quota enabled: none
      quota enabled: none

      Attachments

        Issue Links

          Activity

            [LU-17395] df -h is limited by project quota even if quota enforcement is disabled
            scherementsev Sergey Cheremencev added a comment - - edited

            Looking again at this issue I found a good solution. As indicator of enabled or disabled quota qtype we could return from MDT:

            enabled = !!(glbl_pool->qpi_slv_nr[QMT_STYPE_MDT, qtype] + glbl_pool->qpi_slv_nr[QMT_STYPE_OST, qtype]);
            
            or there is already a function for that:
            
            enabled = !!(qpi_slv_nr(glbl_pool, qtype);

            Another words, at least one connected slave means that appropriate quota type is enabled.

            I believe there is a lot of ways how to return this flag in obd_quotactl. We could even consider to print this flag in general "lfs quota" output.

            This approach gives us ability to avoid walking through the slaves to check whether they enforced or not. This also would help with LU-16660.
            There are 2 problems Im seeing here.

            1. qpi_slv_nr for global pool are incremented at MDS start phase even if exists appropriate directory. I.e. if quota type has been enabled for this slave just once(appropriate directory created), it would be always accounted in qpi_slv_nr(see qmt_pool_prepare->qmt_pool_add). It doesn't look correct and should be fixed. There is already a mechanism to increment newly connected slaves in qmt_pool_new_conn. At first look it is enough to remove qpi_slv_nr changing from qmt_pool_add to increment slaves only in flight when they are connecting.
            2. Another problem of this approach is how to detect that quota type becomes disabled. It is not implemented yet. However, I don't think it is impossible - we could look at qmt_intent_lock(qqi_lockh at qsd side) state - if it is cancelled, we should decrement index for appropriate slave.
            scherementsev Sergey Cheremencev added a comment - - edited Looking again at this issue I found a good solution. As indicator of enabled or disabled quota qtype we could return from MDT: enabled = !!(glbl_pool->qpi_slv_nr[QMT_STYPE_MDT, qtype] + glbl_pool->qpi_slv_nr[QMT_STYPE_OST, qtype]); or there is already a function for that: enabled = !!(qpi_slv_nr(glbl_pool, qtype); Another words, at least one connected slave means that appropriate quota type is enabled. I believe there is a lot of ways how to return this flag in obd_quotactl. We could even consider to print this flag in general "lfs quota" output. This approach gives us ability to avoid walking through the slaves to check whether they enforced or not. This also would help with LU-16660 . There are 2 problems Im seeing here. qpi_slv_nr for global pool are incremented at MDS start phase even if exists appropriate directory. I.e. if quota type has been enabled for this slave just once(appropriate directory created), it would be always accounted in qpi_slv_nr(see qmt_pool_prepare->qmt_pool_add). It doesn't look correct and should be fixed. There is already a mechanism to increment newly connected slaves in qmt_pool_new_conn. At first look it is enough to remove qpi_slv_nr changing from qmt_pool_add to increment slaves only in flight when they are connecting. Another problem of this approach is how to detect that quota type becomes disabled. It is not implemented yet. However, I don't think it is impossible - we could look at qmt_intent_lock(qqi_lockh at qsd side) state - if it is cancelled, we should decrement index for appropriate slave.
            adilger Andreas Dilger added a comment - - edited

            In LU-16771 there was also a performance issue related to an application getting the project quota repeatedly due to many statfs() syscalls, and no cache for the project quota on the clients. I think it is reasonable to cache the "quota enabled/disabled" state on the client and MDS, along with the quota usage for an ID+type for a short time (same as statfs), as long as these are not used to make actual decisions about IO. For these cases, the values are mostly cosmetic, so caching them for a few seconds is fine.

            There would need to be a tunable like "{llite,osc}.*.enable_quota_cache=0" or "llite.*.quota_cache_seconds=0" to disable the cache for sanity-quota to work properly when fetching the quota, or in case of problems in the field.

            adilger Andreas Dilger added a comment - - edited In LU-16771 there was also a performance issue related to an application getting the project quota repeatedly due to many statfs() syscalls, and no cache for the project quota on the clients. I think it is reasonable to cache the "quota enabled/disabled" state on the client and MDS, along with the quota usage for an ID+type for a short time (same as statfs), as long as these are not used to make actual decisions about IO. For these cases, the values are mostly cosmetic, so caching them for a few seconds is fine. There would need to be a tunable like "{ llite,osc}.*.enable_quota_cache=0 " or " llite.*.quota_cache_seconds=0 " to disable the cache for sanity-quota to work properly when fetching the quota, or in case of problems in the field.
            scherementsev Sergey Cheremencev added a comment - - edited

            The solution of this problem could be common with LU-16660. In my case I also wanted to know whether quotas enforced or not. The problem is that to be 100% sure that quotas are disabled we need to ask all "quota slaves", i.e. all OSTs and MDTs. It is too expensive to ask all targets, so probably it is worth to have some common flag on MDS0 that is set to "TRUE" only when quota on all targets is disabled(quota_slave=none). 

            scherementsev Sergey Cheremencev added a comment - - edited The solution of this problem could be common with LU-16660 . In my case I also wanted to know whether quotas enforced or not. The problem is that to be 100% sure that quotas are disabled we need to ask all "quota slaves", i.e. all OSTs and MDTs. It is too expensive to ask all targets, so probably it is worth to have some common flag on MDS0 that is set to "TRUE" only when quota on all targets is disabled(quota_slave=none). 

            It seems like a bug that statfs() is affected if project quota is not enforced. Is there some way that ll_statfs_project()->quotactl_ioctl() can determine that quota is not enforced?

            adilger Andreas Dilger added a comment - It seems like a bug that statfs() is affected if project quota is not enforced. Is there some way that ll_statfs_project()->quotactl_ioctl() can determine that quota is not enforced?

            People

              wc-triage WC Triage
              aboyko Alexander Boyko
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: