[LU-3322] ko2iblnd support for different map_on_demand and peer_credits between systems

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0
• Labels: patch
• Severity: 3
• Bugzilla ID: 20,543
• Assignee: Amir Shehata (Inactive)
• Reporter: Jeremy Filizetti

Description

ko2iblnd currently doesn't support different values of peer_credits or map_on_demand between systems.

After I finish some testing I will upload a patch to gerrit in the next couple of days.

Issue Links

• is duplicated by: LU-5718 RDMA too fragmented with router
• is duplicated by: LU-7401 OOM after LNet initialization with not default peer_creadits on mlx5
• is related to: LU-7351 LNet router crash during bring up of infiniband interface.
• is related to: LU-7650 ko2iblnd map_on_demand can't negotitate when page sizes are different between nodes.
• is related to: LU-7314 In kiblnd_rejected(), NULL pointer 'cp' may be passed to function and can be dereferenced there
• is related to: LU-12419 ppc64le: "LNetError: RDMA has too many fragments for peer_ni" when reading two files
• is related to: LUDOC-286 Document the effects of LU-3322
• is related to: LU-7101 Lnet: Support per NI map-on-demand
• is related to: LU-7569 IB leaf switch caused LNet routers to crash
• is related to: LU-5783 o2iblnd: investigate new memory registration mechanisms

Comments

[Joseph Gmitter (Inactive)]

http://review.whamcloud.com/17074/ has landed for 2.8.
Resolving the ticket as noted in the commentary above.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/17074/
Subject: LU-3322 lnet: make connect parameters persistent
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4c689a573fafcfa1ca7474a275f958e00b1deddc

[Doug Oucharek (Inactive)]

This ticket seems to have expanded to be a catch-all for anything related to map_on_demand/peer_credit settings. I'd rather see this ticket be used for its original purpose, what Jeremy describes above. Anything new should become a new ticket (or set of tickets) so we don't get confused and link a bunch of new tickets to this one believing these patches will solve all issues in this area.

Once patch http://review.whamcloud.com/#/c/17074/ has landed, I'd like this ticket to be closed. If there are any more problems with optimized settings for specific hardware setups, please open separate tickets so they can be prioritized and addressed accordingly.

[Jeremy Filizetti]

I don't know anything about the ko2iblnd-opa, I've always used a custom module parameter file for Lustre. Now that you've included the link I see this is now being included with Lustre which I wasn't aware of before. From what I can see you should be ok to use those parameters with those adapters but I'm not sure they are the "ideal" settings.

[Chris Hunter (Inactive)]

Hi Jeremy,
Thanks for the explanation at least it shows the challenges involved. From your comments the default ko2iblnd-opa parameters (ie. LU-6735) should work for ConnectX[123] and Truescale adapters.
Our hardware, we have max work requests per QP (max_qp_wr) vaules of 16351, 16383 or 16384.

[Jeremy Filizetti]

I only used stock centos 6 kernels for the initial work. Only after noting that MLX5 memory registration does not support FMR have I even started looking at and testing Mellanox ofed. There are not any issues with this patch and map_on_demand settings that I'm aware of even though they seem to be getting reported here as such. The problem is that it requires too much low level understanding of the driver to configure and ko2iblnd does not abstract the differing hardware well enough at this point. My goal with this patch was to allow interop with systems configured for IB WAN performance and those that may come from a vendor solution with different parameters. Given the current memory registration upstream changes and lack of flexibility, ko2iblnd really needs some additional work to make things more robust and support multiple configurations. This patch only serves as a stop-gap for that larger necessary work. The best that really can be done is to make recommendations to people based on their needs here.

[Chris Hunter (Inactive)]

There are different bundles of the OFED/infiniband kernel drivers available from different sources eg) openfabrics.org OFED, mellanox OFED (MOFED), Truescale OFED+ and RHEL rdma. All these packages seem to tweak the kernel drivers (eg. different versions, additional code, etc.) To further confuse the list, there a multiple versions of the packages (eg. MOFED 2.x, 3.x).

Which IB driver packaging are you using for testing the ko2iblnd patches ? stock OFED ? MOFED ?

thanks,
chris hunter

[Dmitry Eremin (Inactive)]

The following parameters are compatible with mlx5 and works fine for OPA:

options ko2iblnd-opa peer_credits=62 peer_credits_hiw=64 credits=1024 concurrent_sends=62 ntx=2048 map_on_demand=256 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1

[Olaf Weber (Inactive)]

FWIW 18446744073709551503 = -113 assuming 2's complement. On x86 Linux, think that's
also -EHOSTUNREACH.

[Jeremy Filizetti]

If you are still using your previous configuration you could modify things to make everything work.

1. Server with mlx5 doesn't support FMR so map_on_demand should remain 0.
2. Router can support FMR with the mlx4 (not sure about hfi) so you could use map_on_demand=256. You will need to drop your concurrent_sends to <=62 I believe for the QP to not fail creation for the mlx4.
3. Client with hfi I don't know if it supports FMR. But you could use defaults (map_on_demand=0) if not.

With all that I think your configuration will be able to connect to everything. At least from what I quickly glanced over with respect to the map_on_demand settings.