[LU-10875] Optimize o2iblnd wrq size

Fri, 6 Apr 2018 22:37:05 +0000

Description written by Mike Marciniszyn

The issue grows out of the issues:

Lustre creates a send queue size of 16193 with our default module parameters
- For example GPFS is two orders of magnitude smaller
- https://ibbugzilla.ph.intel.com/bugzilla/show_bug.cgi?id=142066#c1 contains an analysis of the sizing
- The size is based on the number of credits (default 128) and the LNET_MAX_IOV (hardcoded 256)
  - The LNET_MAX_IOV is the key driver to the large size
  - https://ibbugzilla.ph.intel.com/bugzilla/show_bug.cgi?id=142507#c37 contains the formula
- Lustre’s above sizing does not consider conns_per_peer
  - I’m pretty sure credits are global regardless of the number of conns_per_peer and the QP size should be split evenly on the parallel QPs
- The TID code pre-allocates the flow structures for each send wqe

The key here is that the LNET_MAX_IOV sizes the QP based on the anticipated number of WQEs to consume a credit.

Modern Lustre memory region use (FMR,FRMR*) uses much fewer WQEs to consume a credit. Worse case 3 (vs. 256).

It is possible to detour the memory foot print by setting hfi1 max_qp_wrs to 4095. This will reduce the credits available to Lustre to 8, and will probably impact parallelism on the receive side.

Whamcloud Community JIRA

[LU-10875] Optimize o2iblnd wrq size