[LU-10823] max_create_count triggering uneven distribution across OSTs Created: 16/Mar/18 Updated: 20/Jul/18 Resolved: 20/Jul/18 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.10.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor |
| Reporter: | Jesse Stroik | Assignee: | WC Triage |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Centos 7 |
||
| Issue Links: |
|
||||||||
| Severity: | 2 | ||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
I set two OSTs on a file system to max_create_count=0 to facilitate migrating data off of them so I could destroy and recreate the zpools. Each OST has a corrupted spacemap.
The OSTs in question:
iliad-OST0018_UUID 56246673664 21707936640 34538672768 39% /iliad[OST:24] During the process of running lfs_migrate, I found that the very next OST in the list, OST:26 below, was getting a disproportionate number of files assigned to it. I set max_create_count=0 for that OST as well. I ran into the same issue on OST:27. Then OST:28 as soon as setting max_create_count=0 to OST:27.
The file system was previously well balanced and is about 86% used.
iliad-OST0013_UUID 52.4T 46.2T 6.1T 88% /iliad[OST:19]
Last week I upgraded the Metadata / messaging and object store servers from Centos 6 running lustre 2.7 to Centos 7 running lustre 2.10.3 with ZFS 0.7.5. The MDT is still ldiskfs.
I ran a couple of quick tests to adjust the balancing which may help diagnose the issue.
While actively migrating files from OST 24,25 to the rest of the FS, I tested adjusting qos_prio_free=95
$ date; lfs df /iliad | egrep "OST:(12|28)" $ date; lfs df /iliad | egrep "OST:(12|28)" Change for OST12: (49691904768-49690633216) / 1024 = 1241.75 Change for OST28: (51064366592-51048918400) / 1024 = 15086.125
Looks like it's allocating more than ten times the amount you'd expect – worse yet since OST28 is fuller than most. I then set qos_prio_free back to default and set qos_threshold_rr to 100 to attempt to get straight round robin balancing.
$ date; lfs df /iliad | egrep "OST:(12|28)" $ date; lfs df /iliad | egrep "OST:(12|28)"
(49696041856-49691904768) / 1024 = 4040.125 (51070885248-51064366592) / 1024 = 6365.875
That seems to be in the realm of blind round robin for written files.
Perhaps max_create_count isn't taken into account during the balancing algorithm and the file goes to the next available OST in order. In this case, would I be better off deactivating the OSTs? |
| Comments |
| Comment by Taizeng Wu [ 25/Apr/18 ] |
|
We have encounter same problem. Environemnt:
We set some OST (40-49, 60-69) max_count_create=0 early. Yesterday, wo found a OST50 in the high read operations which cause some user access lustre slowly, then execute `lfs df -i` to found that OST50 usage have more than others. I create a test that touch about 160 files, found that OST50 have heen allocated 10 times more than others. Then i set qos_threshold_rr to 50, run test agin, it seems to be work in the round robin for written files. lfs df -h UUID bytes Used Available Use% Mounted on filesystem_summary: 4.6P 573.4T 4.0P 12% /Share
|
| Comment by Andreas Dilger [ 20/Jul/18 ] |
|
Closing this as a duplicate of |