[LU-7522] conf-sanity test_32b: FAIL: verify quota failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Critical
• Reporter: Jian Yu
• Assignee: Niu Yawei (Inactive)
• Resolution: Duplicate
• Severity: 3

Description

While running conf-sanity test 32b with disk2_7-ldiskfs.tar.bz2 from http://review.whamcloud.com/16663, the test failed as follows:

Disk quotas for user 60000 (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/tmp/t32/mnt/lustre
                  30720*      0   20480       -       3*      0       2       -
t32fs-MDT0000_UUID
                      0       -       0       -       1       -       0       -
t32fs-MDT0001_UUID
                      0       -       0       -       2       -       0       -
t32fs-OST0000_UUID
                  30720       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0
bspace, act:30720, exp:10240
 conf-sanity test_32b: @@@@@@ FAIL: verify quota failed

Maloo report: https://testing.hpdd.intel.com/test_sets/3acf93b6-9bb5-11e5-9930-5254006e85c2

The quota verification failure also occurred with disk2_7-zfs.tar.bz2:
https://testing.hpdd.intel.com/test_sets/cf518162-9b78-11e5-b907-5254006e85c2
https://testing.hpdd.intel.com/test_sets/409ef834-9b76-11e5-b7e5-5254006e85c2

Issue Links

• is related to: LU-7192 conf-sanity test_32c: ZFS test failur...

Comments

[Peter Jones]

NIu

Could you please look into this failure?

Thanks

Peter

[Jian Yu]

The same tests passed against disk2_4-*.tar.bz2.

[Niu Yawei (Inactive)]

After upgrading, looks Lustre generated two files for quota_usr on MDT1 automatically.

Before upgrading:

t32fs-MDT0001_UUID
                      0       -       0       -       0       -       0       -

After upgrading:

t32fs-MDT0001_UUID
                      0       -       0       -       2       -       0       -

Could you find out which files were generated? (mount the MDT1 as ldiskfs, then find the files owned by quota_usr) Thanks.

[Jian Yu]

Hi Niu,

After upgrading, on MDT0000:

# mount -t ldiskfs -o loop /tmp/t32/mdt /tmp/t32/mnt/mdt
# cd /tmp/t32/mnt/mdt
# ls -lR | grep quota_usr
-rw-r--r-- 1 quota_usr quota_usr    0 Dec  8 02:25 t32_qf_old

On MDT0001:

# mount -t ldiskfs -o loop /tmp/t32/mdt2 /tmp/t32/mnt/mdt1
# cd /tmp/t32/mnt/mdt1
# ls -lR | grep quota_usr
-rw-r--r-- 1 quota_usr quota_usr    0 Dec  8 02:25 t32_qf_old
-rw-r--r-- 1 quota_usr quota_usr    0 Dec  8 02:25 t32_qf_old

So, there are two more t32_qf_old files on MDT0001 after upgrading.

[Niu Yawei (Inactive)]

I checked the original image from http://review.whamcloud.com/#/c/16663/ , it does have 2 more quota_usr files on mdt1 and two more quota_usr objects on ost, looks the image is incorrect in first place.

The objects on ost:

/mnt/ost1/O/200000400/d21/55
/mnt/ost1/O/200000400/d10/1ca
/mnt/ost1/O/0/d21/53

They are all 10M size.

The files on mdt1:

/mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x319:0x0/[0x280000401:0x1:0x0]:0/t32_qf_old
/mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x1:0x0/t32_qf_old

[Jian Yu]

You're right, Niu.

There is a defect in test_32newtarball() with MDSCOUNT=2. After gathering the quota data, t32_qf_old was copied to remote_dir and striped_dir_old directories separately. So, there are 3 t32_qf_old files with user quota_usr. I fixed that defect and generated the disk image again (please refer to the attached disk2_7-ldiskfs.tar.bz2).

However, before upgrading:

Disk quotas for user 60000 (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
     /mnt/t32fs   30708       0   40960       -       3       0       4       -
t32fs-MDT0000_UUID
                      0       -       0       -       1       -       0       -
t32fs-MDT0001_UUID
                      0       -       0       -       2       -       0       -
t32fs-OST0000_UUID
                  30708       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0

After upgrading:

Disk quotas for user 60000 (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
/tmp/t32/mnt/lustre
                  30724       0   40960       -       3       0       4       -
t32fs-MDT0000_UUID
                      0       -       0       -       1       -       0       -
t32fs-MDT0001_UUID
                      0       -       0       -       2       -       0       -
t32fs-OST0000_UUID
                  30724       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0

bspace, act:30724, exp:30708
 conf-sanity test_32b: @@@@@@ FAIL: verify quota failed

"ls -lR /tmp/t32/mnt/lustre | grep quota_usr" showed that:

-rw-r--r--  1 quota_usr quota_usr 10485760 Dec  9 07:41 t32_qf_old
-rw-r--r-- 1 quota_usr quota_usr 10485760 Dec  9 07:41 t32_qf_old
-rw-r--r-- 1 quota_usr quota_usr 10485760 Dec  9 07:41 t32_qf_old

The kbytes values didn't match.

[Niu Yawei (Inactive)]

The original kbytes 30708 is less than actual size 30720, probably this value is obtained before all dirty flush back? Could you post the patch which fixing test_32newtarball()?

[Jian Yu]

While generating the disk image, after adding "sync" before getting the quota data, disk quotas for user quota_usr became:

Disk quotas for user 60000 (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
     /mnt/t32fs   30724       0   40960       -       3       0       4       -
t32fs-MDT0000_UUID
                      0       -       0       -       1       -       0       -
t32fs-MDT0001_UUID
                      0       -       0       -       2       -       0       -
t32fs-OST0000_UUID
                  30724       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0

I'm fixing test_32newtarball() in http://review.whamcloud.com/16663. Let's close this ticket as a duplicate of LU-7192.

[Niu Yawei (Inactive)]

Could you check the ost image to see where the extra 4 bytes comes from?

[Jian Yu]

Hi Niu,

# mount -t ldiskfs -o loop ost /mnt/ost1
# find ./ -uid 60000 -gid 60000 -exec ls -l {} \;
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d10/1ca
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d21/55
-rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/0/d21/53

Not sure where the extra 4 bytes comes from.

[Niu Yawei (Inactive)]

Ok, now we understand that the extra 4 bytes come from metadata of an object (in remote directory). When creating the image, sometimes metadata wasn't flushed, that's why we sometimes get 30724, sometimes get 30720.