[LU-9158] sanity-quota, test_12b: rebalancing failed Created: 24/Feb/17  Updated: 09/Apr/18  Resolved: 06/Feb/18

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.10.0, Lustre 2.11.0
Fix Version/s: Lustre 2.11.0, Lustre 2.10.4

Type: Bug Priority: Minor
Reporter: James Casper Assignee: Hongchao Zhang
Resolution: Fixed Votes: 0
Labels: None
Environment:

trevis-46vm1/2/3/7/8, Full Group test w/DNE,
RHEL7.3/zfs, branch master, v2.9.53.19.gdad9ce4, b3534

Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

https://testing.hpdd.intel.com/test_sets/5fc29140-f92a-11e6-aa39-5254006e85c2

From test_log:

Disk quotas for user quota_usr (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /mnt/lustre       0       0       0       - 18446744073709551615       0       0       -
lustre-MDT0000_UUID
                      0       -       0       - 18446744073709551615       -       0       -
lustre-MDT0001_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0002_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0003_UUID
                      0       -       0       -       0       -       0       -


 Comments   
Comment by Sarah Liu [ 20/Jun/17 ]

https://testing.hpdd.intel.com/test_sets/94ae4c52-53f0-11e7-bacf-5254006e85c2

CMD: trevis-55vm3,trevis-55vm7 lctl set_param -n osd*.*MDT*.force_sync=1
CMD: trevis-55vm8 lctl set_param -n osd*.*OS*.force_sync=1
Write to ost1 after space freed from ost0...
running as uid/gid/euid/egid 60000/60000/60000/60000, groups:
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d12a.sanity-quota/f12a.sanity-quota-1] [count=17] [oflag=sync]
dd: error writing '/mnt/lustre/d12a.sanity-quota/f12a.sanity-quota-1': Disk quota exceeded
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00329015 s, 0.0 kB/s
Disk quotas for usr quota_usr (uid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /mnt/lustre    3972       0   22528       -       1       0       0       -
lustre-MDT0000_UUID
                      2       -       0       -       1       -       0       -
lustre-MDT0001_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0002_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0003_UUID
                      0       -       0       -       0       -       0       -
lustre-OST0000_UUID
                      0       -   18432       -       -       -       -       -
lustre-OST0001_UUID
                   3971       -    4096       -       -       -       -       -
lustre-OST0002_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0003_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0004_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0005_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0006_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0007_UUID
                      0       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 22528
Files for user (quota_usr):
  File: '/mnt/lustre/d12a.sanity-quota/f12a.sanity-quota-1'
  Size: 0         	Blocks: 7941       IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115205322843896  Links: 1
Access: (0644/-rw-r--r--)  Uid: (60000/quota_usr)   Gid: (60000/quota_usr)
Access: 2017-06-17 17:21:00.000000000 +0000
Modify: 2017-06-17 17:31:24.000000000 +0000
Change: 2017-06-17 17:31:24.000000000 +0000
 Birth: -
Disk quotas for grp quota_usr (gid 60000):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
    /mnt/lustre    3972       0       0       -       1       0       0       -
lustre-MDT0000_UUID
                      2       -       0       -       1       -       0       -
lustre-MDT0001_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0002_UUID
                      0       -       0       -       0       -       0       -
lustre-MDT0003_UUID
                      0       -       0       -       0       -       0       -
lustre-OST0000_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0001_UUID
                   3971       -       0       -       -       -       -       -
lustre-OST0002_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0003_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0004_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0005_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0006_UUID
                      0       -       0       -       -       -       -       -
lustre-OST0007_UUID
                      0       -       0       -       -       -       -       -
Total allocated inode limit: 0, total allocated block limit: 0
Files for group (quota_usr):
  File: '/mnt/lustre/d12a.sanity-quota/f12a.sanity-quota-1'
  Size: 0         	Blocks: 7941       IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d	Inode: 144115205322843896  Links: 1
Access: (0644/-rw-r--r--)  Uid: (60000/quota_usr)   Gid: (60000/quota_usr)
Access: 2017-06-17 17:21:00.000000000 +0000
Modify: 2017-06-17 17:31:24.000000000 +0000
Change: 2017-06-17 17:31:24.000000000 +0000
 Birth: -
error: can't find id for name: quota_usr
Display disk usage and limits.
usage: quota [-q] [-v] [-h] [-o <obd_uuid>|-i <mdt_idx>|-I <ost_idx>]
             [<-u|-g|-p> <uname>|<uid>|<gname>|<gid>|<projid>] <filesystem>
       quota [-o <obd_uuid>|-i <mdt_idx>|-I <ost_idx>] -t <-u|-g|-p> <filesystem>
Files for project (quota_usr):
Invalid project ID: quota_usr sanity-quota test_12a: @@@@@@ FAIL: rebalancing failed 
  Trace dump:

besides the failure, there is also a script problem from the above log. In test_12a/b, it passes quota_usr to quota_error/quota_scan, quota_scan uses it as project ID which is why shows the "cannot find id" error

Comment by Gerrit Updater [ 12/Sep/17 ]

Wei Liu (wei3.liu@intel.com) uploaded a new patch: https://review.whamcloud.com/28957
Subject: LU-9158 test: Use project ID for project quota for quota_scan
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4325e0c368a671b4569c0023b1c5509aa6308b7a

Comment by Gerrit Updater [ 16/Oct/17 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/28957/
Subject: LU-9158 test: Use project ID for project quota for quota_scan
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3a2f24fefbc8c4b6e98dfd6ba9957326ea0e7a97

Comment by Peter Jones [ 16/Oct/17 ]

Landed for 2.11

Comment by Gerrit Updater [ 16/Oct/17 ]

Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/29626
Subject: LU-9158 test: Use project ID for project quota for quota_scan
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: a20168542dc64bf79b7aa099e9db18694aa30b2b

Comment by Gerrit Updater [ 25/Oct/17 ]

John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/29626/
Subject: LU-9158 test: Use project ID for project quota for quota_scan
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 22e5b8c0f605db8af01c6f21718bcac370510580

Comment by James Casper [ 15/Nov/17 ]

Still seeing this issue (2.10.55 b3667).

Patch merged to master on Oct 16.
https://testing.hpdd.intel.com/test_sessions/2296a0e9-443c-4a4f-be3c-878e5b0afad4
was run on Nov 13.

Comment by Peter Jones [ 14/Dec/17 ]

Hongchao

Can you please look into this one?

Thanks

Peter

Comment by Gerrit Updater [ 28/Dec/17 ]

Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: https://review.whamcloud.com/30660
Subject: LU-9158 quota: release quota ASAP if edquot is set
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 04f73ec5b84f8911a0f2c6021fbe7a867ec64e6d

Comment by Gerrit Updater [ 28/Dec/17 ]

Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: https://review.whamcloud.com/30662
Subject: LU-9158 test: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9c7a14dbcebad74af4475de8eb55cfe2914574b7

Comment by Gerrit Updater [ 08/Jan/18 ]

Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: https://review.whamcloud.com/30765
Subject: LU-9158 quota: adjust quota ASAP
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c088d5b08abc0b9c31cb4825c5803dc6730f2ceb

Comment by Gerrit Updater [ 06/Feb/18 ]

Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/30765/
Subject: LU-9158 quota: adjust quota ASAP
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 70e1532f6216e4ff925bb9a902ec6bd8ea81c084

Comment by Peter Jones [ 06/Feb/18 ]

Landed another fix for 2.11

Comment by Minh Diep [ 23/Feb/18 ]

+1 on b2_10
https://testing.hpdd.intel.com/test_sets/1bd99da6-11f3-11e8-bd00-52540065bddc

Comment by Gerrit Updater [ 23/Feb/18 ]

Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/31404
Subject: LU-9158 quota: adjust quota ASAP
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: d804fb8cb007d321b507fcdbf0e11b359b336e1d

Comment by Gerrit Updater [ 05/Apr/18 ]

John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/31404/
Subject: LU-9158 quota: adjust quota ASAP
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: b03de1fe3d11d65716020cfd07e0db16628af5c7

Generated at Sat Feb 10 02:23:45 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.