Details
-
Improvement
-
Resolution: Unresolved
-
Medium
-
None
-
Lustre 2.14.0, Lustre 2.17.0
-
3
-
9223372036854775807
Description
The lfs-setquota.1 man page should document the minimum quota allocation units (qunits) for inodes (1024) and blocks (1MiB) for each server, and recommend quota limits that allocate at least tens of qunits per target in order to minimize the effect of "stranding" unused inodes/space on servers when a qunit is partially used on one target and cannot be moved to another target.
For example, creating a number of files and then reducing/enforcing the quota limits leaves a system that has no available qunits to allocate to some OSTs:
# for M in {0..3}; do lfs mkdir -i $M -c 1 /mnt/lustre/dir$M; done
# lfs setquota -u quota -I 2068 -i 2048 /mnt/lustre
./runas -u quota_usr touch /mnt/lustre/dir0/f{1..3000}
./runas -u quota_usr touch /mnt/lustre/dir2/f{1..3000}
# lctl set_param -P osd*.*.quota_slave.enabled=u
# lfs quota -vu quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/lustre 0 0 0 - 6000* 2048 2068 -
lustre-MDT0000_UUID 0 - 0 - 3000* - 3000 -
lustre-MDT0001_UUID 0 - 0 - 0 - 0 -
lustre-MDT0002_UUID 0 - 0 - 3000* - 3000 -
lustre-MDT0003_UUID 0 - 0 - 0 - 0 -
Even after removing almost all files it is still impossible to create new files on MDT0001 and MDT0003 because they have no qunits that can be allocated to them, even though the total number of files in the filesystem is below the user's quota limit:
# ./runas -u quota_usr touch /mnt/lustre/dir3/f{3300.3500}; done
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
[touch] [/mnt/lustre/dir3/f3300]
touch: cannot touch ‘/mnt/lustre/dir3/f3300’: Disk quota exceeded
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
[touch] [/mnt/lustre/dir3/f3301]
touch: cannot touch ‘/mnt/lustre/dir3/f3301’: Disk quota exceeded
running as uid/gid/euid/egid 1000/1000/1000/1000, groups: 1000
:
:
rm -f /mnt/lustre/dir1/f{5..3000}
rm -f /mnt/lustre/dir3/f{3..3000}
# lfs quota -vu quota_usr /mnt/lustre
Disk quotas for usr quota_usr (uid 1000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/lustre 0 0 0 - 6 2048 2068 -
lustre-MDT0000_UUID 0 - 0 - 4 - 952 -
lustre-MDT0001_UUID 0 - 0 - 0 - 0 -
lustre-MDT0002_UUID 0 - 0 - 2 - 952 -
lustre-MDT0003_UUID 0 - 0 - 0 - 0 -
