[LU-7958] Directory Operation Performance Degradation on DNE

Thu, 7 Apr 2016 15:54:41 +0000

We have been performing on going testing of DNE in Lustre 2.8. During testing, we have noticed that severe performance degradation occurs when directory operations involving 2 or more MDSs occur. We are using mdtest to assess performance of our setup. We are using directories which are stripped a varying number of MDSs (all have a starting index of 0) to vary the MDS count. Our setup contains 4 MDSs, 4 OSSs, and an independent MGS. Each server is backed by zfs and each server has a pool built with 4 solid state drives. Below is a summary of results from mdtest. I've attached the raw output just in case as well (Catalyst-mdtest-3-25-2016.txt).

Directory creation:

MDS Count	Max	Min	Mean	Std Dev
1	5430.605	3665.080	4601.181	647.529
2	140.306	108.522	126.983	11.541
3	105.811	96.243	102.489	3.741
4	109.556	103.074	106.963	2.482

Directory removal:

MDS Count	Max	Min	Mean	Std Dev
1	2667.779	2126.894	2326.902	206.254
2	163.166	86.190	107.369	32.263
3	61.288	57.917	59.675	1.438
4	81.529	62.300	69.171	7.785

Directory stat:

MDS Count	Max	Min	Mean	Std Dev
1	53112.492	52023.002	52549.901	483.261
2	20169.405	20068.291	20118.116	44.012
3	18580.815	17621.591	18308.662	397.660
4	16904.242	16836.548	16863.872	24.812

We checked some output from testing back in October 2015. A similar trend of degradation is present, though the performance degradation did not appear to be as severe. For this test, we were running 4 MDSs each w/ 1 MDT, 2 OSSs each w/ 1 OST, and clients were running the same lustre version as the servers. Backend was zfs. I attached the logs from that test as well (Zwicky-mdtest-10-7-2015.txt).

Let us know if you need any other information or more testing.

Whamcloud Community JIRA

[LU-7958] Directory Operation Performance Degradation on DNE