[LU-4428] performance-sanity test_5: mknod(f59999) error: Disk quota exceeded

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.5.1, Lustre 2.4.3, Lustre 2.5.3, Lustre 2.10.2, Lustre 2.10.4, Lustre 2.10.5, Lustre 2.10.7, Lustre 2.12.1
• Labels: zfs
• Environment: Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/5/; FSTYPE=zfs; ; OSTCOUNT=2; OSTSIZE=8388608
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Jian Yu

Description

performance-sanity test_5 failed as follows:

Total disk size: 16257024  block-softlimit: 16258048 block-hardlimit: 17070950 inode-softlimit: 55785 inode-hardlimit: 58574
<~snip~>
===== mdsrate-lookup-1dir.sh Test preparation: creating 63715 files.
+ /usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt 'f%%d'
+ chmod 0777 /mnt/lustre
drwxrwxrwx 9 root root 99840 Apr 14  2107 /mnt/lustre
+ su mpiuser sh -c "/usr/lib64/openmpi/bin/mpirun -mca boot ssh -machinefile /tmp/mdsrate-lookup-1dir.machines -np 2 /usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt 'f%%d' "
<~snip~>
rank 1: mknod(f59999) error: Disk quota exceeded
rank 0: mknod(f58728) error: Disk quota exceeded

Maloo report: https://maloo.whamcloud.com/test_sets/fc281616-73c4-11e3-b4ff-52540035b04c

Issue Links

• is duplicated by: LU-5146 Test failure performance-sanity test_5: readv failed: Connection reset by peer

Comments

[Niu Yawei (Inactive)]

inode-softlimit: 55785 inode-hardlimit: 58574

The hardlimit was set to 58574.

/usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt 'f%%d'

However, mdsrate-lookup-1dir.sh want to create 63715 files.

test-framework.sh set hardlimit based on total inodes for whole cluster. (see setup_quota() ), but mdsrate-lookup-1dir.sh creates files based on free inodes on MDT. (see inodes_available()). I think they'd be unified.

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/61/
FSTYPE=zfs

The same failure occurred: https://maloo.whamcloud.com/test_sets/50a6f7f0-ebe5-11e3-82b2-52540035b04c

[Peter Jones]

Nathaniel

Could you please look into this one?

Thanks

Peter

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_5/47/
FSTYPE=zfs

The same failure occurred:
https://maloo.whamcloud.com/test_sets/99e3e576-cb70-11e3-95c9-52540035b04c

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/73/ (2.4.3 RC1)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs

https://maloo.whamcloud.com/test_sets/e70e3c7c-ac60-11e3-81d7-52540035b04c

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/39/ (2.5.1 RC1)
Distro/Arch: RHEL6.5/x86_64
FSTYPE=zfs

The same failure occurred:
https://maloo.whamcloud.com/test_sets/7a0a591c-a607-11e3-8a1b-52540035b04c

[Jian Yu]

More instance on Lustre b2_5 branch:
https://maloo.whamcloud.com/test_sets/6355c6e8-7a15-11e3-bce8-52540035b04c
https://maloo.whamcloud.com/test_sets/249915d8-7ec3-11e3-a0a8-52540035b04c
https://maloo.whamcloud.com/test_sets/8728f446-9ac7-11e3-96fb-52540035b04c