[LU-3322] ko2iblnd support for different map_on_demand and peer_credits between systems

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0
• Labels: patch
• Severity: 3
• Bugzilla ID: 20,543
• Assignee: Amir Shehata (Inactive)
• Reporter: Jeremy Filizetti

Description

ko2iblnd currently doesn't support different values of peer_credits or map_on_demand between systems.

After I finish some testing I will upload a patch to gerrit in the next couple of days.

Issue Links

• is duplicated by: LU-5718 RDMA too fragmented with router
• is duplicated by: LU-7401 OOM after LNet initialization with not default peer_creadits on mlx5
• is related to: LU-7351 LNet router crash during bring up of infiniband interface.
• is related to: LU-7650 ko2iblnd map_on_demand can't negotitate when page sizes are different between nodes.
• is related to: LU-7314 In kiblnd_rejected(), NULL pointer 'cp' may be passed to function and can be dereferenced there
• is related to: LU-12419 ppc64le: "LNetError: RDMA has too many fragments for peer_ni" when reading two files
• is related to: LUDOC-286 Document the effects of LU-3322
• is related to: LU-7101 Lnet: Support per NI map-on-demand
• is related to: LU-7569 IB leaf switch caused LNet routers to crash
• is related to: LU-5783 o2iblnd: investigate new memory registration mechanisms

Comments

[Joseph Gmitter (Inactive)]

http://review.whamcloud.com/17074/ has landed for 2.8.
Resolving the ticket as noted in the commentary above.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/17074/
Subject: LU-3322 lnet: make connect parameters persistent
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4c689a573fafcfa1ca7474a275f958e00b1deddc

[Doug Oucharek (Inactive)]

This ticket seems to have expanded to be a catch-all for anything related to map_on_demand/peer_credit settings. I'd rather see this ticket be used for its original purpose, what Jeremy describes above. Anything new should become a new ticket (or set of tickets) so we don't get confused and link a bunch of new tickets to this one believing these patches will solve all issues in this area.

Once patch http://review.whamcloud.com/#/c/17074/ has landed, I'd like this ticket to be closed. If there are any more problems with optimized settings for specific hardware setups, please open separate tickets so they can be prioritized and addressed accordingly.

[Jeremy Filizetti]

I don't know anything about the ko2iblnd-opa, I've always used a custom module parameter file for Lustre. Now that you've included the link I see this is now being included with Lustre which I wasn't aware of before. From what I can see you should be ok to use those parameters with those adapters but I'm not sure they are the "ideal" settings.

[Chris Hunter (Inactive)]

Hi Jeremy,
Thanks for the explanation at least it shows the challenges involved. From your comments the default ko2iblnd-opa parameters (ie. LU-6735) should work for ConnectX[123] and Truescale adapters.
Our hardware, we have max work requests per QP (max_qp_wr) vaules of 16351, 16383 or 16384.

[Jeremy Filizetti]

I only used stock centos 6 kernels for the initial work. Only after noting that MLX5 memory registration does not support FMR have I even started looking at and testing Mellanox ofed. There are not any issues with this patch and map_on_demand settings that I'm aware of even though they seem to be getting reported here as such. The problem is that it requires too much low level understanding of the driver to configure and ko2iblnd does not abstract the differing hardware well enough at this point. My goal with this patch was to allow interop with systems configured for IB WAN performance and those that may come from a vendor solution with different parameters. Given the current memory registration upstream changes and lack of flexibility, ko2iblnd really needs some additional work to make things more robust and support multiple configurations. This patch only serves as a stop-gap for that larger necessary work. The best that really can be done is to make recommendations to people based on their needs here.