[LU-5718] RDMA too fragmented with router

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0, Lustre 2.9.0
• Labels: llnl
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: Johann Lombardi (Inactive)

Description

Got an IOR failure on the soak cluster with the following errors:

Oct  7 21:54:01 lola-23 kernel: LNetError: 3613:0:(o2iblnd_cb.c:1134:kiblnd_init_rdma()) RDMA too fragmented for 192.168.1.115@o2ib100 (256): 128/256 src 128/256 dst frags
Oct  7 21:54:01 lola-23 kernel: LNetError: 3618:0:(o2iblnd_cb.c:428:kiblnd_handle_rx()) Can't setup rdma for PUT to 192.168.1.114@o2ib100: -90
Oct  7 21:54:01 lola-23 kernel: LNetError: 3618:0:(o2iblnd_cb.c:428:kiblnd_handle_rx()) Skipped 7 previous similar messages

Liang told me that this is a known issue with routing. That said, the IOR process is not killable and the only option is to reboot the client node. We should at least fail "gracefully" by returning the error to the application.

Issue Links

• duplicates: LU-7385 Bulk IO write error
• duplicates: LU-3322 ko2iblnd support for different map_on_demand and peer_credits between systems
• is related to: LU-9420 Bad Check slipped into repo
• is related to: LU-10252 backport change LU-5718 change 12451/12 to b2_8_fe
• is related to: LU-7401 OOM after LNet initialization with not default peer_creadits on mlx5
• is related to: LUDOC-378 Document wrq_sge as an o2iblnd parameter
• is related to: LU-12419 ppc64le: "LNetError: RDMA has too many fragments for peer_ni" when reading two files
• is related to: LU-7210 ASSERTION( peer->ibp_connecting == 0 )
• is related to: LU-7569 IB leaf switch caused LNet routers to crash
• is related to: LU-7650 ko2iblnd map_on_demand can't negotitate when page sizes are different between nodes.

Comments

[Cory Spitz]

LUDOC-378 is linked to this issue.

[Cory Spitz]

Looks like we should have opened a LUDOC ticket to document wrq_sge.

[Stanford Research Computing Center]

Ah! Thanks for the clarification, Chris and Doug! I was a bit lost as the parameters changed along the work done in this ticket. We'll test this right away.
All the best,
Stephane

[Doug Oucharek (Inactive)]

Your router needs wrq_sge=2.

[Chris Horn]

You need to set wrq_sge=2 on the routers, too.

[Stephane Thiell]

Hi,
Could you please explain what is required to make the patches that landed work? We have tried 2.9 FE + patches from both LU-5718 and LU-9420 but are still seeing the problem on the routers. We have set wrq_sge=2 on the clients, and let the default wrq_sge=1 on the routers. We were not able to patch the servers at the moment (running IEEL3), see DELL-221.

on the router with wrq_sge=1 (10.210.34.213@o2ib1 is an OSS not patched):

[ 1111.504575] LNetError: 8688:0:(o2iblnd_cb.c:1093:kiblnd_init_rdma()) RDMA has too many fragments for peer 10.210.34.213@o2ib1 (256), src idx/frags: 128/147 dst idx/frags: 128/147
[ 1111.522352] LNetError: 8688:0:(o2iblnd_cb.c:430:kiblnd_handle_rx()) Can't setup rdma for PUT to 10.210.34.213@o2ib1: -90

Clients and routers are using mlx5, servers are using mlx4.

Thanks,
Stephane