Major Description
We have been performing on going testing of DNE in Lustre 2.8. During testing, we have noticed that severe performance degradation occurs when directory operations involving 2 or more MDSs occur. We are using mdtest to assess performance of our setup. We are using directories which are stripped a varying number of MDSs (all have a starting index of 0) to vary the MDS count. Our setup contains 4 MDSs, 4 OSSs, and an independent MGS. Each server is backed by zfs and each server has a pool built with 4 solid state drives. Below is a summary of results from mdtest. I've attached the raw output just in case as well (Catalyst-mdtest-3-25-2016.txt).
Directory creation:
| MDS Count | Max | Min | Mean | Std Dev |
|---|---|---|---|---|
| 1 | 5430.605 | 3665.080 | 4601.181 | 647.529 |
| 2 | 140.306 | 108.522 | 126.983 | 11.541 |
| 3 | 105.811 | 96.243 | 102.489 | 3.741 |
| 4 | 109.556 | 103.074 | 106.963 | 2.482 |
Directory removal:
| MDS Count | Max | Min | Mean | Std Dev |
|---|---|---|---|---|
| 1 | 2667.779 | 2126.894 | 2326.902 | 206.254 |
| 2 | 163.166 | 86.190 | 107.369 | 32.263 |
| 3 | 61.288 | 57.917 | 59.675 | 1.438 |
| 4 | 81.529 | 62.300 | 69.171 | 7.785 |
Directory stat:
| MDS Count | Max | Min | Mean | Std Dev |
|---|---|---|---|---|
| 1 | 53112.492 | 52023.002 | 52549.901 | 483.261 |
| 2 | 20169.405 | 20068.291 | 20118.116 | 44.012 |
| 3 | 18580.815 | 17621.591 | 18308.662 | 397.660 |
| 4 | 16904.242 | 16836.548 | 16863.872 | 24.812 |
We checked some output from testing back in October 2015. A similar trend of degradation is present, though the performance degradation did not appear to be as severe. For this test, we were running 4 MDSs each w/ 1 MDT, 2 OSSs each w/ 1 OST, and clients were running the same lustre version as the servers. Backend was zfs. I attached the logs from that test as well (Zwicky-mdtest-10-7-2015.txt).
Let us know if you need any other information or more testing.