[LU-7293] DNE2 perfomance analysis

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Affects Version/s: Lustre 2.9.0
• Labels: dne2 dne3
• Environment: DNE2 system with up to 16 MDS servers. Uses up to 400 client nodes spread across 20 physical nodes. All the results are based on mdtest 1.9.3 runs.

Epic/Theme

Performance

• Assignee: Di Wang (Inactive)
• Reporter: James A Simmons

Description

I did a detail study of the client scaling behavior for 10k and 100k files per directory using 1,2,4, and 8 MDS servers each having one MDT. I also attempted to collect dat for 16 MDS servers but the results were so bad I didn't both to finish collecting them since it would take several months to finish the 16 node case.

Issue Links

• is related to: LU-6831 The ticket for tracking all DNE2 bugs

Comments

[Peter Jones]

closing ancient ticket

[Jian Yu]

Hi James,

Thank you very much for the report!

[James A Simmons]

Here is the final report of our results for our DNE2 performance analysis

http://info.ornl.gov/sites/publications/Files/Pub59510.pdf

Enjoy the read. Perhaps we can link it to the wiki. If people want it linked to the wiki we can do that
and then close this ticket. If not we can keep this ticket open for a few more weeks so people can
have a chance to read this.

[James A Simmons]

Oh this doesn't look right.

ldlm.namespaces.sultan-MDT0000-mdc-ffff8803f3d12c00.lru_size=29
ldlm.namespaces.sultan-MDT0001-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0002-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0003-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0004-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0005-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0006-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0007-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0008-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT0009-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000a-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000b-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000c-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000d-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000e-mdc-ffff8803f3d12c00.lru_size=0
ldlm.namespaces.sultan-MDT000f-mdc-ffff8803f3d12c00.lru_size=0

[Di Wang (Inactive)]

Hmm, according to the slab information in Nov 15th, it seems "lustre_inode_cache" is much more than "inode_cache", so it means client has more ll_inode_info than inode, hmm, maybe ll_inode_info is leaked somewhere. Do you still keep that client? Could you please get lru_size for me?

lctl get_param ldlm.*.*MDT*.lru_size

[James A Simmons]

I did both file only operations and with directory operations. I did tracked down the issue of directory operations as well. What is happening in that case is that the lustre inode cache is consuming all the memory on the client thus causing various timeout and client evictions and reconnects. This only happens for when many directory operations are performed. When only doing file operations the memory pressure issues go away. My latest testings as all been without the --index.