[LU-7650] ko2iblnd map_on_demand can't negotitate when page sizes are different between nodes.

• Type: Bug
• Resolution: Duplicate
• Priority: Major
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: ko2iblnd lnet ppc
• Environment: Power8 client nodes running Ubuntu 14.04 and Lustre servers running RHE6.6 with all running the latest pre-2.8 lustre stack.
• Severity: 3
• Assignee: Amir Shehata (Inactive)
• Reporter: James A Simmons

Description

With the mlx5 driver support mostly complete I have been testing on our Power8 nodes the latest 2.8 stack plus some ko2iblnd patches to make it work. With the work from LU-5783 I can now set the peer_credits to 63 and using the patch from LU-7101 set the map_on_demand to 16 pages. The problem is that the Power8 clients can't connect to our servers which have map_on_demand set to 256 pages. The reason is the set test to validate the connection compares the page counts on each side. On Power8 16 pages equals 256 pages on x86 platforms so comparing page count is an invalid test.

Issue Links

• is related to: LU-8573 IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc->bd_md_count == 0 ) failed
• is related to: LU-10157 LNET_MAX_IOV hard coded to 256
• is related to: LU-5718 RDMA too fragmented with router
• is related to: LU-10129 map-on-demand set to 32 doesn't work on OPA
• is related to: LU-12419 ppc64le: "LNetError: RDMA has too many fragments for peer_ni" when reading two files
• is related to: LU-10300 Can the Lustre 2.10.x clients support 64K kernel page?
• is related to: LU-3322 ko2iblnd support for different map_on_demand and peer_credits between systems

Comments

[James A Simmons]

This is now a duplicate of LU-10157

[Peter Jones]

James

Doug is focused on some of the upstream changes in the coming weeks so will not get to this in the 2.10 timeframe

Peter

[James A Simmons]

Doug do you have time to take another shot at this?

[Doug Oucharek (Inactive)]

Is there any good "architecture" style document about this API? I don't find header files very useful in understanding "how" the API should be used.

[James A Simmons]

Actually I just got a copy of the RHEL7.3 source and the new RDMA generic api is there. So the answer is the solution is ready for todays customers.

[Doug Oucharek (Inactive)]

James: how far back does the new RDMA RW api go (kernel version)? My understanding is the new api starts about Linux 4.6 which is very far out for current Lustre users. We need to band-aid ko2iblnd to keep going for the next few years.

I agree that ko2iblnd is going to need a rewrite to stay relevant for the upstream kernel.

[James A Simmons]

With the new RDMA RW api and map_on_demand being broken not only on Power8 but in general it looks like a big rewrite of the ko2ibllnd driver is needed

[Peter Jones]

Yep. We'll need to take another run at this.

[James A Simmons]

Appears support for Power8 broke things.

[Peter Jones]

Landed for 2.9