[LU-10129] map-on-demand set to 32 doesn't work on OPA

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.11.0
• Type: Bug
• Priority: Critical
• Reporter: Amir Shehata (Inactive)
• Assignee: Amir Shehata (Inactive)
• Resolution: Fixed
• Labels: llnl
• Severity: 3

Description

With patch https://review.whamcloud.com/29290/, map-on-demand set to 32 doesn't work. map-on-demand needs to be set to 256 in order for large transfers to complete.

Issue Links

• is related to: LU-7650 ko2iblnd map_on_demand can't negotita...
• is related to: LU-10157 LNET_MAX_IOV hard coded to 256

Comments

[Olaf Faaland]

We do have OPA fabrics, so I've added the llnl and topllnl labels to this ticket based on the following comment in LU-10089:

One thing to note, if you're using OPA you should use map-on-demand set to 256. I'm still analyzing this issue and hopefully will have a patch soon. This issue is tracked under LU-10129

I haven't seen unexplained symptoms on our OPA-connected nodes.

What would I likely see?  disconnects/reconnects?  Failed BRW operations?

[Amir Shehata (Inactive)]

At the LND layer RDMA writes fail. So that could translate to RPC failures/bulk write failures, and other FS issues.

[Amir Shehata (Inactive)]

Here is a summary of my investigation on map-on-demand, and a proposal for a change to resolve the issues around this. I'd like feedback to see if I missed something:
https://wiki.hpdd.intel.com/display/LNet/o2iblnd+map_on_demand

[James A Simmons]

While are looking to fix this maybe its time to revist LU-7650.

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29995
Subject: LU-10129 lnd: rework map_on_demand behavior
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d6fb47b0e3cc1521f8f33dcc7dd78a9abcb27154

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30309
Subject: LU-10129 lnd: set device capabilities
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 200fb9081989add33457554a221ecfe087da1176

[Chris Hunter (Inactive)]

Does hfi1 module parameter "num_user_contexts" have to match the LND map_on_demand setting ?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30309/
Subject: LU-10129 lnd: set device capabilities
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 70a73c52d5a780e580e49ccfa778c4beab340c9c

[Olaf Faaland]

Do this patch and 29290 need to be backported to 2.10?  I'm not requesting it, just asking the question.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/29995/
Subject: LU-10129 lnd: rework map_on_demand behavior
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2c38e8f5b08e7475ffc37915fe0300505a5759db

[Peter Jones]

Landed for 2.11. Flagged for consideration for 2.10.x until Amir is back in the office and able to comment

[Amir Shehata (Inactive)]

These list of patches don't need to be ported to 2.10.x. They were instigated by:

LU-9983 ko2iblnd: allow for discontiguous fragments

which is not in 2.10

Di committed his changed

LU-9983 osp: align the OSP request size by 4k

which should hide the initial problem of discontigous fragments.

[Chris Hunter (Inactive)]

LU-9500 proposes similar fix for fragment alignment.