[LU-11678] sanity-quota test 1 fails with 'user write success, but expect EDQUOT'

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.3
• Affects Version/s: Lustre 2.12.0, Lustre 2.13.0, Lustre 2.12.1
• Labels: DNE
• Environment: DNE
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: James Nunez (Inactive)

Description

sanity-quota test_1 is failing in a DNE configuration, review-dne-part-4, with

'user write success, but expect EDQUOT' 

meaning that the user write, dd, should not have succeeded.

Looking at the failure at https://testing.whamcloud.com/test_sets/43baf332-e791-11e8-b67f-52540065bddc , the last lines seen in the client test_log is

Disk quotas for usr quota_usr (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /mnt/lustre   10240*      0   10240       -       1       0       0       -
lustre-MDT0000_UUID
                      0       -       0       -       1       -       0       -
lustre-MDT0001_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0002_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0003_UUID
                      0       -       0       -       0       -       0       -
lustre-OST0000_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0001_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0002_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0003_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0004_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0005_UUID
                  10240*      -   10240       -       -       -       -       -
lustre-OST0006_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0007_UUID
                      0       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 10240
uid 60000 is using default file quota setting
Files for user (quota_usr):
  File: '/mnt/lustre/d1.sanity-quota/f1.sanity-quota-0'
  Size: 11534336  	Blocks: 20480      IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144119902892982281  Links: 1
Access: (0644/-rw-r--r--)  Uid: (60000/quota_usr)   Gid: (60000/quota_usr)
Access: 2018-11-13 20:03:03.000000000 +0000
Modify: 2018-11-13 20:03:07.000000000 +0000
Change: 2018-11-13 20:03:07.000000000 +0000
 Birth: -
 sanity-quota test_1: @@@@@@ FAIL: user write success, but expect EDQUOT 

There are no errors or anything to indicate that there was a problem in any of the console or dmesg logs.

This may be related to or the same as LU-5245. Although this test failed in October several times with the same error message, these test failures have many errors in the logs.

It looks like this test started failing again on 2018-11-13.

Logs for other recent failures are at
https://testing.whamcloud.com/test_sets/52c657cc-e8d6-11e8-86c0-52540065bddc
https://testing.whamcloud.com/test_sets/9608d2d6-e79f-11e8-bfe1-52540065bddc
https://testing.whamcloud.com/test_sets/b126d406-e889-11e8-b67f-52540065bddc

Issue Links

• is duplicated by: LU-9609 sanity-quota test_1: project write success, but expect edquot
• is related to: LU-9326 sanity-quota test_2: group create success, but expect EDQUOT
• is related to: LU-12128 sanity-quota test_1: user write success, but expect EDQUOT
• is related to: LU-11687 allow "lfs fid2path" on open-unlinked files
• is related to: LU-5245 sanity-quota test_1: user write success, but expect EDQUOT
• is related to: LU-11390 sanity-quota test_61: test timeout
• is related to: LU-11425 Support quota for DoM

Comments

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34627
Subject: LU-11678 tests: Additional debug
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b0987312838e2fd5ef47e09457ea94a4f3450e05

[Patrick Farrell (Inactive)]

Seeing some very similar failures on test 2...
[-https://testing.whamcloud.com/test_sets/14a7deb0-5349-11e9-92fe-52540065bddc-]

My bad, this was LU-12100.

[James Nunez (Inactive)]

sanity-quota test 1 is still failing. LU-12128 was opened due to new test failures. So, reopening this ticket and closing LU-12128 as a duplicate.

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33747/
Subject: LU-11678 quota: protect quota flags at OSC
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 77d9f4e05a5c366ad0f7c2e97a338c6958676f73

[Patrick Farrell (Inactive)]

Hitting this on b2_12 as well.

[Sebastien Buisson]

Also hit with 'group' instead of 'user':
https://testing.whamcloud.com/test_sets/b9fa2b58-49c0-11e9-a256-52540065bddc

[Patrick Farrell (Inactive)]

Additionally hit this with 'project' instead of user:
https://testing.whamcloud.com/test_sessions/bdd56f63-a386-4074-87bc-e9f8f9f4f768

[Hongchao Zhang]

Hi, Andreas,
The problem fixed by the patch 33747 exists before the patch from LU-11390, the problem is easily triggered because
the pending over quota flag is not sent back long time before the quota is really over quota after the patch
from LU-11390 was landed, which means the minor disorder of the replies can't trigger the issue.

[Andreas Dilger]

Hongchao, does your patch 33747 fix the problem related to the patch from LU-11390, or is another patch needed?

[Hongchao Zhang]

Hi Andres,
It could be related to the patch in LU-11390, which modified the condition to check the pending over-quota flag and this issue
is just caused by the improper processing of this flag.