Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
Lustre 2.16.0
-
3
-
9223372036854775807
Description
sanity-quota test_1a, 1b, and 1c failed on 2.16.0 RC1 full-zfs-part-2 test session:
https://testing.whamcloud.com/test_sets/0d7fdd5a-b404-4b82-bd32-aaf538cee475
Disk quotas for grp quota_usr (gid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/lustre 10245* 0 10240 - 1 0 0 -
lustre-MDT0000 2 - 0 - 1 - 0 -
lustre-OST0000 0 - 0 - - - - -
lustre-OST0001 0 - 0 - - - - -
lustre-OST0002 0 - 0 - - - - -
lustre-OST0003 10244 - 0 - - - - -
lustre-OST0004 0 - 0 - - - - -
lustre-OST0005 0 - 0 - - - - -
lustre-OST0006 0 - 0 - - - - -
Total allocated inode limit: 0, total allocated block limit: 0
Files for group (quota_usr), count=1:
File: /mnt/lustre/d1a.sanity-quota/f1a.sanity-quota-1
Size: 11534336 Blocks: 20487 IO Block: 4194304 regular file
Device: 2c54f966h/743766374d Inode: 144117285664063501 Links: 1
Access: (0644/-rw-r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr)
Access: 2024-10-01 10:01:26.000000000 +0000
Modify: 2024-10-01 10:01:36.000000000 +0000
Change: 2024-10-01 10:01:36.000000000 +0000
Birth: 2024-10-01 10:01:26.000000000 +0000
sanity-quota test_1a: @@@@@@ FAIL: user write success, but expect EDQUOT
Attachments
Issue Links
- is related to
-
-
- Open
-
- mentioned in
-
Page Loading...
-
-
-
-
-
-
-
-
-
-
-
-
-
Activity
hongchao.zhang, as Alex pointed out, there are now quite a lot (155) sanity-quota test_71a failures on master:
https://testing.whamcloud.com/search?status%5B%5D=FAIL&test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&start_date=2025-04-08&end_date=2025-04-14&source=sub_tests#redirect
These look like they mostly started on 2025-04-10 after this patch landed to master, except for 21 more failures on 2025-04-04 when the patch was included in master-next.
# MDSCOUNT=2 ONLY=71a bash sanity-quota.sh ... == sanity-quota test 71a: Check PFL with quota pools ===== 18:44:23 (1744483463) Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 User quota (block hardlimit:100 MB) Creating new pool qpool1 Pool lustre.qpool1 created Adding targets to pool OST lustre-OST0000_UUID added to pool lustre.qpool1 OST lustre-OST0001_UUID added to pool lustre.qpool1 Creating new pool qpool2 Pool lustre.qpool2 created Adding targets to pool OST lustre-OST0001_UUID added to pool lustre.qpool2 -u quota_usr space:0 used 0 Write... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0] [count=10] [oflag=direct] dd: error writing '/mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0': Disk quota exceeded 10+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.148691 s, 63.5 MB/s Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 9216 0 102400 - 1 0 0 - lustre-MDT0000 0 - 0 - 1 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 9216 - 10240 - 1 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 10240 Files for user (quota_usr), count=1: File: /mnt/lustre/d71a.sanity-quota/f71a.sanity-quota-0 Size: 9437184 Blocks: 18432 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381366533 Links: 1 Access: (0644/-rw-r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2025-04-12 18:44:31.000000000 +0000 Modify: 2025-04-12 18:44:31.000000000 +0000 Change: 2025-04-12 18:44:31.000000000 +0000 Birth: - sanity-quota test_71a: @@@@@@ FAIL: user write failure, but expect success
with this patch landed sanity-quota/71a doesn't pass locally, I did a bisection:
COMMIT TESTED PASSED FAILED COMMIT DESCRIPTION 2f96622a52 5 0 5 BAD LU-18293 test: use direct IO for quota test 1811e77562 5 5 0 GOOD LU-16134 utils: un-deprecate 'lctl set_param -F'
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58533/
Subject: LU-18293 test: use direct IO for quota test
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2f96622a522924c7b9e26fd2dfdf34eb6227b7a6
"Hongchao Zhang <hongchao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58533
Subject: LU-18293 test: use direct IO for quota test
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 43184902d29ed8ed053a5da38e70781398b65f19
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57078
Subject: LU-18293 test: check to sync quota setting
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: a69b054a22e1ae97509b659b086338b5286241fc
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56628/
Subject: LU-18293 test: check to sync quota setting
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a9caa85ad2e344ff7ea2a2f33367290600439a13
Note that there is a new test failure rate report that can be used to see the change in subtest failure rates over time, even if a subtest is already failing intermittently:
https://testing.whamcloud.com/reports?test_set_script_id=61149410-4a46-11e0-a7f6-52540025f9af&sub_test_script_id=91a1b9eb-e7d9-491e-9cdb-b6ccf3a4a53a&source=fail_rate_trend#redirect
This link is showing sanity-quota test_73 going from 0% failure rate to over 75% failure rate in the past few days. The previous failure spikes in March and January were caused by patches that were failing almost all of the subtests.