[LU-5718] RDMA too fragmented with router

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0, Lustre 2.9.0
• Labels: llnl
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: Johann Lombardi (Inactive)

Description

Got an IOR failure on the soak cluster with the following errors:

Oct  7 21:54:01 lola-23 kernel: LNetError: 3613:0:(o2iblnd_cb.c:1134:kiblnd_init_rdma()) RDMA too fragmented for 192.168.1.115@o2ib100 (256): 128/256 src 128/256 dst frags
Oct  7 21:54:01 lola-23 kernel: LNetError: 3618:0:(o2iblnd_cb.c:428:kiblnd_handle_rx()) Can't setup rdma for PUT to 192.168.1.114@o2ib100: -90
Oct  7 21:54:01 lola-23 kernel: LNetError: 3618:0:(o2iblnd_cb.c:428:kiblnd_handle_rx()) Skipped 7 previous similar messages

Liang told me that this is a known issue with routing. That said, the IOR process is not killable and the only option is to reboot the client node. We should at least fail "gracefully" by returning the error to the application.

Issue Links

• duplicates: LU-7385 Bulk IO write error
• duplicates: LU-3322 ko2iblnd support for different map_on_demand and peer_credits between systems
• is related to: LU-9420 Bad Check slipped into repo
• is related to: LU-10252 backport change LU-5718 change 12451/12 to b2_8_fe
• is related to: LU-7401 OOM after LNet initialization with not default peer_creadits on mlx5
• is related to: LUDOC-378 Document wrq_sge as an o2iblnd parameter
• is related to: LU-12419 ppc64le: "LNetError: RDMA has too many fragments for peer_ni" when reading two files
• is related to: LU-7210 ASSERTION( peer->ibp_connecting == 0 )
• is related to: LU-7569 IB leaf switch caused LNet routers to crash
• is related to: LU-7650 ko2iblnd map_on_demand can't negotitate when page sizes are different between nodes.