[LU-1720] Quota doesn't work over 4TB on single OST

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.1.4, Lustre 1.8.9
• Affects Version/s: Lustre 1.8.8
• Environment: CentOS5.8 Lustre-1.8.8-wc1
• Severity: 2
• Assignee: Yang Sheng
• Reporter: Shuichi Ihara (Inactive)

Description

We set quota "ug3" to all OSTs and MDT, then an also set 5TB quota limitation to a user. But, if user1 writes files to single OST, it exceeds quota limitation when total file size gets 4TB.

# lfs quota -v -u user1 /lustre/
Disk quotas for user user1 (uid 1000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /lustre/ 4295057504       0 5368709120       -      13       0       0       -
lustre-MDT0000_UUID
                      4       -    1024       -      13       -       0       -
lustre-OST0000_UUID
                      0       -    1024       -       -       -       -       -
lustre-OST0001_UUID
                4295057500*      - 4294959104       -       -       -       -       -
lustre-OST0002_UUID
                      0       -    1024       -       -       -       -       -
lustre-OST0003_UUID
                      0       -    1024       -       -       -       -       -
..
..

# lctl get_param lquota.*.quota_type
lquota.lustre-OST0001.quota_type=ug3
lquota.lustre-OST0004.quota_type=ug3
lquota.lustre-OST0008.quota_type=ug3
lquota.lustre-OST000c.quota_type=ug3
lquota.lustre-OST0011.quota_type=ug3
lquota.lustre-OST0015.quota_type=ug3
lquota.lustre-OST0019.quota_type=ug3
lquota.lustre-OST001d.quota_type=ug3
lquota.lustre-OST0021.quota_type=ug3
lquota.lustre-OST0025.quota_type=ug3
lquota.lustre-OST0028.quota_type=ug3
lquota.lustre-OST002d.quota_type=ug3
lquota.lustre-OST0031.quota_type=ug3
lquota.lustre-OST0035.quota_type=ug3
lquota.lustre-OST0039.quota_type=ug3

# lctl get_param lquota.*.quota_type
lquota.lustre-MDT0000.quota_type=ug3

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA

Comments

[Yang Sheng]

Patch landed to all branch. Close bug.

[Peter Jones]

Landed to b1_8. Still needs to land to 2.x branches

[Shuichi Ihara (Inactive)]

Hi, I've confirmed the patch LU-1720 solves this problem, would you please merge if the review is finished?

[Shuichi Ihara (Inactive)]

I reproduced this problem with lustre-1.8.8 on RHEL5 and confirmed it's fixed by kernel patch on LU-1720. Also confirmed fixing on lustre-2.1.2 with RHEL5 and the problem doesn't happen on RHEL6 even without patches.

[Yang Sheng]

patch for b1_8: http://review.whamcloud.com/3599

[Yang Sheng]

patch for b2_1: http://review.whamcloud.com/3600

[Niu Yawei (Inactive)]

The patch updated incorrectly when supporting new kernel. Please check: http://review.whamcloud.com/#change,3599

[Shuichi Ihara (Inactive)]

Niu, thanks for analysis. what do you mean mis-updated?

[Niu Yawei (Inactive)]

Thanks for your update, Ihara!

Finally, I found the reason: the kernel quota-large-limits-rhel5.patch (which makes kernel do_set_dqblk() support 64bits) is mis-updated in ba5dd769f66194a80920cf93d6014c78729efaae (LU-674 kernel update RHEL5.7 [2.6.18-274.3.1.el5]).

Yangshen, could you take a look on this, and fix the patch? Thanks.

[Shuichi Ihara (Inactive)]

This is reproducer of this problem.
And here is what I did to make the lustre and tesing.

# MDS
# mkfs.lustre --reformat --mgs --mdt --param mdt.quota_type=ug3 /dev/sdb1
# mount -t lustre /dev/sdb1 /mnt/lustre/MDT
# lctl get_param lquota.*.quota_type
lquota.mdd_obd-lustre-MDT0000.quota_type=ug3


# OSS
# mkfs.lustre --reformat --ost --mgsnode=192.168.100.129@o2ib --param ost.quota_type=ug3 /dev/mapper/LUN59
# mount -t lustre /dev/mapper/LUN59 /mnt/lustre/LUN59
# lctl get_param lquota.*.quota_type
lquota.lustre-OST0000.quota_type=ug3


# Client
# mount -t lustre 192.168.100.129@o2ib:/lustre /lustre
# nohup /tmp/reproducer.sh &

# lfs quota -u user1 -v /lustre/
Disk quotas for user user1 (uid 1000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
       /lustre/ 4295204904       0 5368709120       -     100       0       0       -
lustre-MDT0000_UUID
                      0       -    1024       -     100       -       0       -
lustre-OST0000_UUID
                4295204904*      - 4294966272