[LU-6864] DNE3: Support multiple modify RPCs in flight for MDT-MDT connection

• Type: Technical task
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Labels: dne3 patch performance recovery
• Assignee: Hongchao Zhang
• Reporter: Gregoire Pichon

Description

This feature is a complement to the LU-5319 that implements the support of multiple modify RPCs in flight for MDC-MDT connection.

It will improve the performance of modify metadata cross-MDT operations, while ensuring the correctness of recovery handling (request resend, and MDT recovery).

Issue Links

• is related to: LU-14761 DNE2 Metadata degradation
• is related to: LU-16065 replay-single test_81a: rm remote dir failed
• is related to: LU-16126 Lustre 2.15.51 mdtest fails with MPI_Abort errors while adjusting max_rpcs_in_progress and using large number of clients
• is related to: LU-5319 Support multiple slots per client in last_rcvd file
• is related to: LU-17821 LMR1c: Improve DNE Distributed Transaction Performance
• is related to: LU-9436 DNE2 - performance improvement with wide stripping directory
• is related to: LU-11999 DNE performance improvement
• is related to: LU-6753 Fix several minor improvements to multislots feature
• is related to: LU-12125 Allow parallel rename of regular files

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47656/
Subject: LU-6864 tests: properly skip sanity/245b in interop
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c4ebdc96061ae9c24ac471b2866f2087bc3e98d4

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47656
Subject: LU-6864 tests: properly skip sanity/245b in interop
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 91fa883754380b935b92d6e01fe5b1063483cc0c

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/14375/
Subject: LU-6864 osp: manage number of modify RPCs in flight
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 23028efcae01bf1274a68fd2dd379fbb33300e82

[Gerrit Updater]

"Lai Siyao <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46735
Subject: LU-6864 osp: manage number of modify RPCs in flight
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dce4b2800030f981b91249227daee40966ae4afd

[Andreas Dilger]

James, thanks for testing this out. The graphs would be much easier to compare if they had the before/after results for the same tests on a single graph.

As for max_mod_rpcs_in_flight for the MDS, I think it is totally reasonable to increase this higher than 8, since we don't want clients to bottleneck on the MDSes if they are busy handling lots of client requests. The MDS has max_rpcs_in_flight=512, so it wouldn't be unreasonable to have the MDS tune this to match mds.MDS.mdt_out.threads_max.

[James A Simmons]

https://www.opensfs.org/wp-content/uploads/Evaluation-of-DoM-SNE-scaling_Simmons_revised051821.pdf

Looking at the data compared to my LUG talk since it is the same hardware we see improvements in file removals and stats. I do see a regression in reads of small files. I don't know if that is due to recent changes or the patch itself at this point. This is using the new default settings of max_mod_rpcs_in_flight = 8. I need to bump it up to see if I can get more out of it.

[James A Simmons]

The 14375 patch has been revived. Currently I'm collecting data to see if it helps with DNE2 scaling issues.