Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
None
-
3
-
9223372036854775807
Description
Recent testing has shown that increasing the default round-robin depth for directories created near the root directory from the current default of 3 to some higher depth (5/6) or larger can improve performance for larger filesystems.
However, care must be taken to avoid adding unnecessary overhead for systems with only a small number of MDTs, because creating a remote directory can take up to 10x longer to finish.
It may be that the ideal situation is to tune the max-inherit-rr value set on the root directory based on the number of MDTs formatted in the filesystem. If the number of MDTs is 10+ then the extra overhead for creating remote directories is compensated by the improved performance from getting more MDTs involved. Otherwise, something like "3 + MDT_COUNT" might be reasonable.
The NCSA LUG presentation reports using a default max-inherit-rr=10 with their 48-MDT Delta filesystem to provide good MDT space balance (45/48 MDTs are within 6% usage of each other).
The HRT LUG presentation reports using max-inherit-rr=17 for each of the project directories on their 40-MDT filesystem and maintain MDT balance within 3% usage.
Conversely, we have seen imbalanced MDT usage on smaller clusters using only the max-inherit-rr=3 setting.