[LU-5638] sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn't 0. 1

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.7.0
• Labels: zfs
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for nasf <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/a26efad0-3e95-11e4-916a-5254006e85c2.

The sub-test test_33 failed with the following error:

Used inodes for user 60000 isn't 0. 1

Please provide additional information about the failure here.

Info required for matching: sanity-quota 33

Issue Links

• is related to: LU-9592 sanity-quota test cases 33 remove from ALWAYS_EXCEPT
• is related to: LU-11024 Broken inode accounting of MDT on ZFS
• is related to: LU-2435 inode accounting in osd-zfs is racy
• is related to: LU-7991 Add project quota for ZFS

Comments

[Andreas Dilger]

It appears that this was "fixed" by the landing of https://review.whamcloud.com/27093 which changed the detection of ZFS project quotas but broke detection of ZFS dnode accounting. That patch landed to b2_10 on Dec 20, 2017 (master landing on Nov 9, 2017).

[Hongchao Zhang]

there is no abnormal information in the logs, and it could still be related to the ZFS performance.
this issue has not occurred since Dec 19, 2017

[Peter Jones]

Hongchao

Could you please advise on this one?

Thanks

Peter

[Dilip Krishnagiri (Inactive)]

sanity-quota test 33 is failing. Maloo link to look at needed information

https://testing.hpdd.intel.com/test_sets/8442a52c-7bad-11e7-a168-5254006e85c2

Error: 'Used inode for user 60000 is 1, expected 10'
Wait for setattr on objects finished...
sleep 5 for ZFS OSD
Waiting for local destroys to complete
CMD: onyx-45vm7,onyx-45vm8 lctl set_param -n osd*.MDT.force_sync=1
CMD: onyx-45vm10 lctl set_param -n osd*.OS.force_sync=1
Verify disk usage after write
Verify inode usage after write
sanity-quota test_33: @@@@@@ FAIL: Used inode for user 60000 is 1, expected 10
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:5291:error()
= /usr/lib64/lustre/tests/sanity-quota.sh:2423:test_33()
= /usr/lib64/lustre/tests/test-framework.sh:5567:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:5606:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:5453:run_test()
= /usr/lib64/lustre/tests/sanity-quota.sh:2450:main()
Dumping lctl log to /test_logs/2017-08-07/lustre-reviews-el7-x86_64-custom-1_101_1_49353_-70097615897520-184409/sanity-quota.test_33.*.1502136383.log
CMD: onyx-45vm10,onyx-45vm1.onyx.hpdd.intel.com,onyx-45vm2,onyx-45vm7,onyx-45vm8 /usr/sbin/lctl dk > /test_logs/2017-08-07/lustre-reviews-el7-x86_64-custom-1_101_1_49353_-70097615897520-184409/sanity-quota.test_33.debug_log.\$(hostname -s).1502136383.log;
dmesg > /test_logs/2017-08-07/lustre-reviews-el7-x86_64-custom-1_101_1_49353_-70097615897520-184409/sanity-quota.test_33.dmesg.\$(hostname -s).1502136383.log
Resetting fail_loc on all nodes...CMD: onyx-45vm10,onyx-45vm1.onyx.hpdd.intel.com,onyx-45vm2,onyx-45vm7,onyx-45vm8 lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null
done.

[James Nunez (Inactive)]

It looks like sanity-quota test 33 is still failing with ZFS servers.

Logs for two recent failures are at:
https://testing.hpdd.intel.com/test_sets/506f3d2e-480d-11e7-91f4-5254006e85c2
https://testing.hpdd.intel.com/test_sets/ac775cfe-4a84-11e7-91f4-5254006e85c2

[Gerrit Updater]

Niu Yawei ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27423
Subject: LU-5638 tests: re-enable zfs quota tests
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ec72f9a800214354e537b3f95c83c1ea509fa178

[Niu Yawei (Inactive)]

The new issue is created at LU-9597.

[Niu Yawei (Inactive)]

I think the old issue should have been fixed once the LU-2435 being landed, we can re-enable test_33 for zfs testing now, I'll cook a patch to re-enable it.

The new occurrences on ldiskfs is another issue, I believe it's a defect in project quota:

sanity-quota test_33: @@@@@@ FAIL: Used space for project 1000:18432, expected:20480

I think we'd open a new ticket for it.

[Andreas Dilger]

I don't think skipping the test is the right way forward, except as a short-term workaround. Instead, someone needs to take the time to figure out what file is being left behind with this UID.

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/1637dff4-4839-11e7-bc6c-5254006e85c2

[Bob Glossman (Inactive)]

being seen in non-zfs tests too. example:
https://testing.hpdd.intel.com/test_sets/834ddd66-472e-11e7-b3fe-5254006e85c2

I note that test 33 is skipped with ALWAYS_EXCEPT for test runs on zfs. Maybe it needs to be skipped all the time on everything.