Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Lustre 2.8.0
-
None
-
3
-
9223372036854775807
Description
While running conf-sanity test 32b with disk2_7-ldiskfs.tar.bz2 from http://review.whamcloud.com/16663, the test failed as follows:
Disk quotas for user 60000 (uid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/tmp/t32/mnt/lustre
30720* 0 20480 - 3* 0 2 -
t32fs-MDT0000_UUID
0 - 0 - 1 - 0 -
t32fs-MDT0001_UUID
0 - 0 - 2 - 0 -
t32fs-OST0000_UUID
30720 - 0 - - - - -
Total allocated inode limit: 0, total allocated block limit: 0
bspace, act:30720, exp:10240
conf-sanity test_32b: @@@@@@ FAIL: verify quota failed
Maloo report: https://testing.hpdd.intel.com/test_sets/3acf93b6-9bb5-11e5-9930-5254006e85c2
The quota verification failure also occurred with disk2_7-zfs.tar.bz2:
https://testing.hpdd.intel.com/test_sets/cf518162-9b78-11e5-b907-5254006e85c2
https://testing.hpdd.intel.com/test_sets/409ef834-9b76-11e5-b7e5-5254006e85c2
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
Hi Niu,
# mount -t ldiskfs -o loop ost /mnt/ost1
# find ./ -uid 60000 -gid 60000 -exec ls -l {} \;
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d10/1ca
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d21/55
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/0/d21/53
Not sure where the extra 4 bytes comes from.
Could you check the ost image to see where the extra 4 bytes comes from?
While generating the disk image, after adding "sync" before getting the quota data, disk quotas for user quota_usr became:
Disk quotas for user 60000 (uid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/t32fs 30724 0 40960 - 3 0 4 -
t32fs-MDT0000_UUID
0 - 0 - 1 - 0 -
t32fs-MDT0001_UUID
0 - 0 - 2 - 0 -
t32fs-OST0000_UUID
30724 - 0 - - - - -
Total allocated inode limit: 0, total allocated block limit: 0
I'm fixing test_32newtarball() in http://review.whamcloud.com/16663. Let's close this ticket as a duplicate of LU-7192.
The original kbytes 30708 is less than actual size 30720, probably this value is obtained before all dirty flush back? Could you post the patch which fixing test_32newtarball()?
You're right, Niu.
There is a defect in test_32newtarball() with MDSCOUNT=2. After gathering the quota data, t32_qf_old was copied to remote_dir and striped_dir_old directories separately. So, there are 3 t32_qf_old files with user quota_usr. I fixed that defect and generated the disk image again (please refer to the attached disk2_7-ldiskfs.tar.bz2).
However, before upgrading:
Disk quotas for user 60000 (uid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/mnt/t32fs 30708 0 40960 - 3 0 4 -
t32fs-MDT0000_UUID
0 - 0 - 1 - 0 -
t32fs-MDT0001_UUID
0 - 0 - 2 - 0 -
t32fs-OST0000_UUID
30708 - 0 - - - - -
Total allocated inode limit: 0, total allocated block limit: 0
After upgrading:
Disk quotas for user 60000 (uid 60000):
Filesystem kbytes quota limit grace files quota limit grace
/tmp/t32/mnt/lustre
30724 0 40960 - 3 0 4 -
t32fs-MDT0000_UUID
0 - 0 - 1 - 0 -
t32fs-MDT0001_UUID
0 - 0 - 2 - 0 -
t32fs-OST0000_UUID
30724 - 0 - - - - -
Total allocated inode limit: 0, total allocated block limit: 0
bspace, act:30724, exp:30708 conf-sanity test_32b: @@@@@@ FAIL: verify quota failed
"ls -lR /tmp/t32/mnt/lustre | grep quota_usr" showed that:
-rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old
The kbytes values didn't match.
I checked the original image from http://review.whamcloud.com/#/c/16663/ , it does have 2 more quota_usr files on mdt1 and two more quota_usr objects on ost, looks the image is incorrect in first place.
The objects on ost:
/mnt/ost1/O/200000400/d21/55 /mnt/ost1/O/200000400/d10/1ca /mnt/ost1/O/0/d21/53
They are all 10M size.
The files on mdt1:
/mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x319:0x0/[0x280000401:0x1:0x0]:0/t32_qf_old /mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x1:0x0/t32_qf_old
Hi Niu,
After upgrading, on MDT0000:
# mount -t ldiskfs -o loop /tmp/t32/mdt /tmp/t32/mnt/mdt # cd /tmp/t32/mnt/mdt # ls -lR | grep quota_usr -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old
On MDT0001:
# mount -t ldiskfs -o loop /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 # cd /tmp/t32/mnt/mdt1 # ls -lR | grep quota_usr -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old
So, there are two more t32_qf_old files on MDT0001 after upgrading.
After upgrading, looks Lustre generated two files for quota_usr on MDT1 automatically.
Before upgrading:
t32fs-MDT0001_UUID
0 - 0 - 0 - 0 -
After upgrading:
t32fs-MDT0001_UUID
0 - 0 - 2 - 0 -
Could you find out which files were generated? (mount the MDT1 as ldiskfs, then find the files owned by quota_usr) Thanks.
Ok, now we understand that the extra 4 bytes come from metadata of an object (in remote directory). When creating the image, sometimes metadata wasn't flushed, that's why we sometimes get 30724, sometimes get 30720.