[LU-10875] Optimize o2iblnd wrq size

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Severity: 3
• Assignee: Sonia Sharma (Inactive)
• Reporter: Amir Shehata (Inactive)

Description

Description written by Mike Marciniszyn

The issue grows out of the issues:

• Lustre creates a send queue size of 16193 with our default module parameters
  • For example GPFS is two orders of magnitude smaller
  • https://ibbugzilla.ph.intel.com/bugzilla/show_bug.cgi?id=142066#c1 contains an analysis of the sizing
  • The size is based on the number of credits (default 128) and the LNET_MAX_IOV (hardcoded 256)
    • The LNET_MAX_IOV is the key driver to the large size
    • https://ibbugzilla.ph.intel.com/bugzilla/show_bug.cgi?id=142507#c37 contains the formula
  • Lustre’s above sizing does not consider conns_per_peer
    • I’m pretty sure credits are global regardless of the number of conns_per_peer and the QP size should be split evenly on the parallel  QPs
  • The TID code pre-allocates the flow structures for each send wqe

 

The key here is that the LNET_MAX_IOV sizes the QP based on the anticipated number of WQEs to consume a credit.

 

Modern Lustre memory region use (FMR,FRMR*) uses much fewer WQEs to consume a credit.   Worse case 3 (vs. 256).

 

It is possible to detour the memory foot print by setting hfi1 max_qp_wrs to 4095.  This will reduce the credits available to Lustre to 8, and will probably impact parallelism on the receive side.