Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-9555

"df /path/to/project" should return projid-specific values

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • Lustre 2.14.0
    • Lustre 2.10.0
    • None
    • 9223372036854775807

    Description

      With local ext4 and XFS filesystems, it is possible to use "df /path/to/directory" to return the current quota usage for the projid associated with that directory as "used", and min(projid quota limit, free space) as "total". This is a natural interface for users/applications, since it represents the used/maximum space for that subdirectory. Otherwise, the user will get EDQUOT back when the project quota runs out for that directory and applications will not be able to figure out how much data they could write into that directory.

      static int ext4_statfs(struct dentry *dentry, struct kstatfs *buf)
      {
              buf->f_type = EXT4_SUPER_MAGIC;
              buf->f_bsize = sb->s_blocksize;
              buf->f_blocks = ext4_blocks_count(es) - EXT4_C2B(sbi, overhead);
              buf->f_bfree = EXT4_C2B(sbi,
                      percpu_counter_sum_positive(&sbi->s_freeclusters_counter) -
                      percpu_counter_sum_positive(&sbi->s_dirtyclusters_counter));
              buf->f_bavail = buf->f_bfree -
                              (ext4_r_blocks_count(es) + resv_blocks);
              if (buf->f_bfree < (ext4_r_blocks_count(es) + resv_blocks))
                      buf->f_bavail = 0;
              buf->f_files = le32_to_cpu(es->s_inodes_count);
              buf->f_ffree = percpu_counter_sum_positive(&sbi->s_freeinodes_counter);
      
      #ifdef CONFIG_QUOTA
              if (ext4_test_inode_flag(dentry->d_inode, EXT4_INODE_PROJINHERIT) &&
                  sb_has_quota_limits_enabled(sb, PRJQUOTA))
                      ext4_statfs_project(sb, EXT4_I(dentry->d_inode)->i_projid, buf);
      #endif
      }
      
      /*
       * Directory tree accounting is implemented using project quotas, where
       * the project identifier is inherited from parent directories.
       * A statvfs (df, etc.) of a directory that is using project quota should
       * return a statvfs of the project, not the entire filesystem.
       * This makes such trees appear as if they are filesystems in themselves.
       */
      void xfs_fill_statvfs_from_dquot(struct kstatfs *statp, struct xfs_dquot *dqp)
      {
              uint64_t                limit;
      
              limit = dqp->q_core.d_blk_softlimit ?
                      be64_to_cpu(dqp->q_core.d_blk_softlimit) :
                      be64_to_cpu(dqp->q_core.d_blk_hardlimit);
              if (limit && statp->f_blocks > limit) {
                      statp->f_blocks = limit;
                      statp->f_bfree = statp->f_bavail =
                              (statp->f_blocks > dqp->q_res_bcount) ?
                               (statp->f_blocks - dqp->q_res_bcount) : 0;
              }
      
              limit = dqp->q_core.d_ino_softlimit ?
                      be64_to_cpu(dqp->q_core.d_ino_softlimit) :
                      be64_to_cpu(dqp->q_core.d_ino_hardlimit);
              if (limit && statp->f_files > limit) {
                      statp->f_files = limit;
                      statp->f_ffree = (statp->f_files > dqp->q_res_icount) ?
                               (statp->f_ffree - dqp->q_res_icount) : 0;
              }
      }
      
      void xfs_qm_statvfs(xfs_inode_t *ip, struct kstatfs *statp)
      {
              if (!xfs_qm_dqget(mp, NULL, xfs_get_projid(ip), XFS_DQ_PROJ, 0, &dqp)) {
                      xfs_fill_statvfs_from_dquot(statp, dqp);
                      xfs_qm_dqput(dqp);
              }
      }
      

      It be useful to be able to do the same with Lustre. One option would be to transfer the projid to the MDS and OSS in the OBD_STATFS request order to get the space usage for that projid directly. Currently the client RPC request body is empty, so there is no place to put the projid, but a new body could be added. The other option would be to check for a projid on the inode in ll_statfs() and ll_obd_statfs(), and if projid != 0 send the equivalent of "lfs quota -p" for that projid, and use it for the statfs "used" and "total" values.

      Attachments

        Issue Links

          Activity

            [LU-9555] "df /path/to/project" should return projid-specific values

            Hello,
            I tried to apply the patch on top of b2_12 but as some prototypes have changed, I'm a bit unsure on how to do it in a good way(tm). Would Whamcloud consider a backport to 2.12? We would LOVE to have this feature on our systems. Thanks for considering it!

            sthiell Stephane Thiell added a comment - Hello, I tried to apply the patch on top of b2_12 but as some prototypes have changed, I'm a bit unsure on how to do it in a good way(tm). Would Whamcloud consider a backport to 2.12? We would LOVE to have this feature on our systems. Thanks for considering it!
            pjones Peter Jones added a comment -

            Landed for 2.14

            pjones Peter Jones added a comment - Landed for 2.14

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36685/
            Subject: LU-9555 quota: df should return projid-specific values
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: e5c8f6670fbeea9ec8c6092dfa4369508da54485

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36685/ Subject: LU-9555 quota: df should return projid-specific values Project: fs/lustre-release Branch: master Current Patch Set: Commit: e5c8f6670fbeea9ec8c6092dfa4369508da54485
            adilger Andreas Dilger added a comment - - edited

            It would also be good to submit a patch to the upstream kernel to include projid into the statx() output. That is tracked under LU-12480.

            adilger Andreas Dilger added a comment - - edited It would also be good to submit a patch to the upstream kernel to include projid into the statx() output. That is tracked under LU-12480 .

            Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36685
            Subject: LU-9555 quota: df should return projid-specific values
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 844d2d2f519c9cffdf10ad637a194bdef1efd3e3

            gerrit Gerrit Updater added a comment - Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36685 Subject: LU-9555 quota: df should return projid-specific values Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 844d2d2f519c9cffdf10ad637a194bdef1efd3e3

            I think to be consistent with ext4 and ZFS the total space would be the quota limit for that projid, the free space is the remaining quota space for that projid, and the available space is based on the quota softlimit, and the same for inodes, something like:

                    if (inode has project limit) {
                            f_blocks = dqb_bhardlimit;
                            f_bfree = min(dqb_bhardlimit - dqb_curspace, f_bfree);
                            f_bavail = min(dqb_bsoftlimit - dqb_curspace, f_bavail);
                            f_files = dqb_ihardlimit;
                            f_ffree = min(dqb_ihardlimit - dqb_curinodes, f_ifree);
                    }
            

            I thought that there is already a way for the "lfs quota" command to print per-OST usage amounts? If there isn't a "granted" field, then it would be possible to just take the unused quota space and divide it evenly across all of the OSTs, so that it adds up to the total unused quota space again at the end.

            adilger Andreas Dilger added a comment - I think to be consistent with ext4 and ZFS the total space would be the quota limit for that projid, the free space is the remaining quota space for that projid, and the available space is based on the quota softlimit, and the same for inodes, something like: if (inode has project limit) { f_blocks = dqb_bhardlimit; f_bfree = min(dqb_bhardlimit - dqb_curspace, f_bfree); f_bavail = min(dqb_bsoftlimit - dqb_curspace, f_bavail); f_files = dqb_ihardlimit; f_ffree = min(dqb_ihardlimit - dqb_curinodes, f_ifree); } I thought that there is already a way for the " lfs quota " command to print per-OST usage amounts? If there isn't a "granted" field, then it would be possible to just take the unused quota space and divide it evenly across all of the OSTs, so that it adds up to the total unused quota space again at the end.

            I got a further questions for 'lfs df' case:

            [root@server_el7_vm1 lustre]# lfs df /mnt/lustre
            UUID 1K-blocks Used Available Use% Mounted on
            lustre-MDT0000_UUID 5825660 47240 5255588 1% /mnt/lustre[MDT:0]
            lustre-OST0000_UUID 9662472 38040 9083760 1% /mnt/lustre[OST:0]
            lustre-OST0001_UUID 9662472 38044 9083756 1% /mnt/lustre[OST:1]
            lustre-OST0002_UUID 9662472 38044 9083756 1% /mnt/lustre[OST:2]
            lustre-OST0003_UUID 9662472 38044 9083756 1% /mnt/lustre[OST:3]

            filesystem_summary: 38649888 152172 36335028 1% /mnt/lustre

            What exactly 'avail' space and total space we should report for each OST/MDT?
            I supposed we might need get each slave's granted space as total space and calculate
            avail space based on that?

            And existed quotactl doesn't report that information back, we might need extend to support that?

            wshilong Wang Shilong (Inactive) added a comment - I got a further questions for 'lfs df' case: [root@server_el7_vm1 lustre] # lfs df /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 5825660 47240 5255588 1% /mnt/lustre [MDT:0] lustre-OST0000_UUID 9662472 38040 9083760 1% /mnt/lustre [OST:0] lustre-OST0001_UUID 9662472 38044 9083756 1% /mnt/lustre [OST:1] lustre-OST0002_UUID 9662472 38044 9083756 1% /mnt/lustre [OST:2] lustre-OST0003_UUID 9662472 38044 9083756 1% /mnt/lustre [OST:3] filesystem_summary: 38649888 152172 36335028 1% /mnt/lustre What exactly 'avail' space and total space we should report for each OST/MDT? I supposed we might need get each slave's granted space as total space and calculate avail space based on that? And existed quotactl doesn't report that information back, we might need extend to support that?
            adilger Andreas Dilger added a comment - - edited

            It should be possible for "lfs df" to pass a new flag like "LL_STATFS_PROJECT" (all the "LL_STATFS_*" flags should be moved into e.g. "enum ll_statfs_flags" to make it easier to find them), which asks the kernel to get the project ID for the inode in ll_obd_statfs(), and then do the quota call from the kernel for each user+OST (which I think is possible).  The user doesn't have to send an explicit project ID from userspace, this can be found in the kernel from the inode.

            No projid-specific handling for "lfs df".

            adilger Andreas Dilger added a comment - - edited It should be possible for " lfs df " to pass a new flag like " LL_STATFS_PROJECT" (all the " LL_STATFS_*" flags should be moved into e.g. " enum ll_statfs_flags " to make it easier to find them), which asks the kernel to get the project ID for the inode in ll_obd_statfs() , and then do the quota call from the kernel for each user+OST (which I think is possible).  The user doesn't have to send an explicit project ID from userspace, this can be found in the kernel from the inode. No projid-specific handling for " lfs df ".

            Andreas, you missed 'lfs df' case.

            wshilong Wang Shilong (Inactive) added a comment - Andreas, you missed 'lfs df' case.

            I don't think we need to get the project quota info from the lower layers of osc_statfs() and mdc_statfs(). Instead, it should get the normal quota information in ll_statfs() via quotactl_ioctl() and then apply limits to osfs returned from ll_statfs_internal() in a similar manner to how ext4_statfs() calls ext4_statfs_project().

            adilger Andreas Dilger added a comment - I don't think we need to get the project quota info from the lower layers of osc_statfs() and mdc_statfs() . Instead, it should get the normal quota information in ll_statfs() via quotactl_ioctl() and then apply limits to osfs returned from ll_statfs_internal() in a similar manner to how ext4_statfs() calls ext4_statfs_project() .

            People

              wshilong Wang Shilong (Inactive)
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: