[LU-17395] df -h is limited by project quota even if quota enforcement is disabled

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Alexander Boyko

Description

$ git log --oneline -n3
d43e8b471f (HEAD -> df_noenf, tag: v2_15_60, tag: 2.15.60, origin/master, origin/HEAD) New tag 2.15.60
54d8859bef LU-17215 tests: sanity/398q to use $tfile
7a56a689d4 LU-17374 gss: get rid of rsi cache entries after req handle
$ MDSCOUNT=2 FSTYPE=ldiskfs bash lustre/tests/llmount.sh
$ lfs project -sp 1212 /mnt/lustre
$ lfs setquota -p 1212 -b 10M -B 10M -i 10K -I 10K /mnt/lustre
$ df -h /mnt/lustre
Filesystem Size Used Avail Use% Mounted on
10.0.2.15@tcp:/lustre 10M 4.0K 10M 1% /mnt/lustre
$ lfs project -d /mnt/lustre
1212 P /mnt/lustre
$ lfs project -sp 0 /mnt/lustre
$ df -h /mnt/lustre
Filesystem Size Used Avail Use% Mounted on
10.0.2.15@tcp:/lustre 612M 3.0M 556M 1% /mnt/lustre
$ lctl get_param osd-ldiskfs.*.quota_slave.info | grep enabled
quota enabled: none
quota enabled: none
quota enabled: none
quota enabled: none

Issue Links

• is related to: LU-16660 don't show grace time if this quota type is disabled
• is related to: LU-15721 projid quota limit statfs() on directories only with PROJINHERIT
• is related to: LU-16771 add client statfs() cache for project quotas
• is related to: LU-9555 "df /path/to/project" should return projid-specific values

Comments

[Sergey Cheremencev]

Looking again at this issue I found a good solution. As indicator of enabled or disabled quota qtype we could return from MDT:

enabled = !!(glbl_pool->qpi_slv_nr[QMT_STYPE_MDT, qtype] + glbl_pool->qpi_slv_nr[QMT_STYPE_OST, qtype]);

or there is already a function for that:

enabled = !!(qpi_slv_nr(glbl_pool, qtype);

Another words, at least one connected slave means that appropriate quota type is enabled.

I believe there is a lot of ways how to return this flag in obd_quotactl. We could even consider to print this flag in general "lfs quota" output.

This approach gives us ability to avoid walking through the slaves to check whether they enforced or not. This also would help with LU-16660.
There are 2 problems Im seeing here.

1. qpi_slv_nr for global pool are incremented at MDS start phase even if exists appropriate directory. I.e. if quota type has been enabled for this slave just once(appropriate directory created), it would be always accounted in qpi_slv_nr(see qmt_pool_prepare->qmt_pool_add). It doesn't look correct and should be fixed. There is already a mechanism to increment newly connected slaves in qmt_pool_new_conn. At first look it is enough to remove qpi_slv_nr changing from qmt_pool_add to increment slaves only in flight when they are connecting.
2. Another problem of this approach is how to detect that quota type becomes disabled. It is not implemented yet. However, I don't think it is impossible - we could look at qmt_intent_lock(qqi_lockh at qsd side) state - if it is cancelled, we should decrement index for appropriate slave.

[Andreas Dilger]

In LU-16771 there was also a performance issue related to an application getting the project quota repeatedly due to many statfs() syscalls, and no cache for the project quota on the clients. I think it is reasonable to cache the "quota enabled/disabled" state on the client and MDS, along with the quota usage for an ID+type for a short time (same as statfs), as long as these are not used to make actual decisions about IO. For these cases, the values are mostly cosmetic, so caching them for a few seconds is fine.

There would need to be a tunable like "{llite,osc}.*.enable_quota_cache=0" or "llite.*.quota_cache_seconds=0" to disable the cache for sanity-quota to work properly when fetching the quota, or in case of problems in the field.

[Sergey Cheremencev]

The solution of this problem could be common with LU-16660. In my case I also wanted to know whether quotas enforced or not. The problem is that to be 100% sure that quotas are disabled we need to ask all "quota slaves", i.e. all OSTs and MDTs. It is too expensive to ask all targets, so probably it is worth to have some common flag on MDS0 that is set to "TRUE" only when quota on all targets is disabled(quota_slave=none). 

[Andreas Dilger]

It seems like a bug that statfs() is affected if project quota is not enforced. Is there some way that ll_statfs_project()->quotactl_ioctl() can determine that quota is not enforced?