[LU-1167] Poor mdtest unlink performance with multiple processes per node

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.1.1
• Environment: SL6.1,; 2.6.32-131.12.1.el6.lustre.20.x86_64
• Severity: 3

Epic

metadata performance

• Assignee: WC Triage
• Reporter: Nathan Rutman

Description

We have noticed in testing that running multiple mdtest processes per node severely degrades unlink performance.

• Lustre mounted once per client; not multimount.
• shared directory case
  This can be seen at a wide range of node counts (5-128) and backends, to varying degrees.
  Interestingly scaling the client count up does not seem to have nearly the same negative performance impact; only the ppn seems to matter.

	total
nodes	jobs	unlink kops
8	8	11.4
8	16	9.7
8	32	8.6
8	64	7

We see the same issue with 1.8.6 clients against the server; we do not see it with 1.8.6 servers.

Issue Links

• is related to: LU-3308 large readdir chunk size slows unlink/"rm -r" performance
• is related to: LU-1695 Demonstrate MDS performance with increasing client load for SMP Affinity

Comments

[Cory Spitz]

It looks as though the comments that were made on 04/Apr/12 were about right. LU-3308 was opened to look into the regression aspect (which is seemingly unrelated to multiple processes per node).

[Mark Mansk]

metabench source

[Cory Spitz]

Bug should be marked as affects all 2.x versions.

[Mark Mansk]

data from metabench runs for 1.8.6,2.3&2.4

[Mark Mansk]

excel sheet with comparisons of 1.8.6 to 2.3 & 2.4 using metabench. show performance drop from 1.8.6 in deletes in same directory for 2.x.

[Cory Spitz]

We could run some apple-to-apples #s on Cray gear between 1.8.6 and 2.1.1. What kind of scale do you need?

[Nathan Rutman]

Cory: yes, but it's a little unclear.
I'm having trouble finding some old performance numbers at scale; I was really hoping Cliff had some older Hyperion numbers using 1.8.x

[Cory Spitz]

If the HW was fixed for the comparison then that doesn't sound like a regression, just a major improvement with different dynamics.

[Nathan Rutman]

Hmm, that's a good point Cory. It gets a little obfuscated with the number of variations in client count, server type, and storage layout. Our 1.8.6 5-client test didn't show the drop with increasing ppn; it remained constant at 5kops. Our 2.1 8-client test did show the decrease, but started from 13kops and went down to 9kops. So it's a little hard to call this a clear regression, when the 2.1 numbers are all above the 1.8 numbers.

[Cory Spitz]

Nathan, I agree with the bottlenecks that you've identified, but I don't think that any of them are regressions. But, maybe I'm wrong wrt c.? In the description you wrote, "We see the same issue with 1.8.6 clients against the server; we do not see it with 1.8.6 servers." Shouldn't we first focus on the regression from 1.8.6 to 2.x?