|
These nodes were RHEL 8.3 based and have been upgraded to RHEL 8.4. At the same time, they got an updated Lustre client with the following OS-compat patches:
- 12d13f33c0 (tag: 2.12.6_9.llnl)
LU-13783 osc: handle removal of NR_UNSTABLE_NFS
- 3cdb219927
LU-12355 llite: MS_* flags and SB_* flags split
- 03e48854cb
LU-12355 llite: totalram_pages changed to atomic_long_t
- b055fe9f7a (tag: 2.12.6_8.llnl)
LU-14690 kernel: new kernel [RHEL 8.4 4.18.0-305.el8]
- 335e03049d (tag: 2.12.6_7.llnl)
LU-14673 sec: annotate algorithms taking optional key
|
|
See also LU-14690
|
|
Serguei
Can you please advise?
Thanks
Peter
|
|
For my reference, my local ticket is TOSS-5228
|
|
Hi Serguei, do you have any update or questions on this? Thanks
|
|
Olaf,
So far it looks like it is possible there's some sort of incompatibility in how ib_post_send is called, but I don't have anything concrete in this direction yet. Do you have any config logs (from building lustre)? I'm not that familiar with Omnipath either. Which version of Omnipath are you using? Basically I want to make sure we end up calling ib_post_send correctly.
Thanks,
Serguei.
|
|
Hi Serguei,
I've attached build.txt - not the config.log, but at least the stdout from ./configure. I'll look into Omnipath info.
thanks,
Olaf
|
|
Hi Olaf,
Here's some more detail regarding what I'd like to try with the OPA build:
lnet/autoconf/lustre-lnet.m4 has a check for ib_post_send() and ib_post_recv() to see if they require const ptr parameters. Could you please try removing this check and build without it? (See attached diff file). I suspect there may be an issue with using the wrong header file when building. The kernel code for 4.18.0 appears to define these functions without the "const" and I think that's what we should be using for OPA, but the stdout you provided indicates that as a result of the configure the "const" version is used.diff.txt
Thanks,
Serguei.
|
|
Hi Serguei,
Sorry, I didn't see your message for some reason. My test was slightly different, but I think still produced the result you need. With the config check sabotaged so HAVE_IB_POST_SEND_RECV_CONST is not defined, the build fails with
My change was
thanks,
Olaf
|
|
Olaf,
I suppose this means we need to look for a different reason why ib_post_send would fail with EINVAL (22) error code.
So far I wasn't able to find the source code for this function provided by Omnipath for RH8.4. Oddly, I also failed to locate Omnipath release notes for RH8.4. Perhaps someone from CN can comment.
Btw, although this is unlikely to be related to this issue, perhaps you want to port LU-13182LU-14690)
Thanks,
Serguei.
|
|
Hi Serguei, I'm told:
"OPA hardware does not have hardware verbs support, so it uses the rdmavt layer to provide software translation. You want to look at rvt_post_send() in drivers/infiniband/sw/rdmavt/qp.c."
We're using the in-kernel OPA driver, so in this case the one in linux-4.18.0-305.el8
thanks
|
|
Olaf,
Have you tried building lustre master and testing if it has the same problem on your RH8.4 OPA machine? There's a chance that one of the o2iblnd patches there may help. Otherwise I'm thinking I'm going to have to add some debug messages to LLNL patch stack so we can use that to get more data.
Thanks,
Serguei.
|
|
Just to document this where it's visible to everyone:
Ran perftest (perftest-4.4-37.0.t4.x86_64) from opal63 (compute), used both opal64 (compute) and opal187 (router) as servers. These are the same nodes, running the same OS, where lnet_selftest fails.
All these tests succeeded:
Those arguments mean:
|
|
Hi Serguei,
Have you tried building lustre master and testing if it has the same problem on your RH8.4 OPA machine?
We were in the process of setting that up when we lost power to the machine. As soon as it's back up, we will test that.
|
|
We've now tried lnet_selftest on the OPA compute nodes opal63 and opal64 using lustre-2.14.0_2.llnl https://github.com/LLNL/lustre/tree/2.14.0_2.llnl. The results were the same as with lustre-2.12.6_9.llnl, that is, the test failed and we saw bad numbers for the results (lots of 0s) and the same error in the kernel dumps as before:
(o2iblnd_cb.c:957:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.5@o2ib18
We're in the process of repeating the same test using lustre master.
|
|
The test with master hit a list corruption BUG, so it was inconclusive. I won't be able to track it down now, I'm leaving for vacation soon and need to prepare.
|
|
Can someone post instructions or a test script so we can take a look? I downloaded the LLNL lustre tarball and built it on a RHEL 8.4 box (stock kernel) and would like to give it a shot, but not very familiar with Lnet self test.
|
|
Dennis,
There's instructions here: https://wiki.lustre.org/LNET_Selftest
Basically, two nodes with LNet running are needed
1) Make sure selftest is loaded on both nodes: modprobe lnet_selftest
2) Make sure LNet is configured on both nodes. Run "lnetctl net show" to list local nids on each.
3) Make the nodes "discover" each other: "lnetctl discover <peer nid>"
4) Copy the wrapper script on one of the nodes. Use primary nids to fill in "TO" and "FROM"
5) Run the script
|
|
The 305 kernel may be the first RHEL kernel with fmr removed...
|
|
Am able to reproduce an EINVAL error internally. Looks like some of the post_send calls work but the ones from kiblnd_sd always fail. Here is a kprobe that I used to dump the return value of rvt_post_send():
r:testprobe rdmavt:rvt_post_send $retval
These threads always succeed:
lst_t_00_24-7492 [039] d... 88520.827471: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 [ko2iblnd] <- rvt_post_send) arg1=0x0
kiblnd_connd-7547 [000] d... 88520.827888: testprobe: (ib_send_mad+0x235/0x420 [ib_core] <- rvt_post_send) arg1=0x0
monitor_thread-7556 [032] d... 88521.503931: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 [ko2iblnd] <- rvt_post_send) arg1=0x0
These threads always fail:
kiblnd_sd_00_01-7549 [014] d... 88520.827678: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 [ko2iblnd] <- rvt_post_send) arg1=0xffffffea
Note 0xffffffea decodes to -22 in decimal.
|
|
The -EINVAL is happening because a post_send of a IB_WR_LOCAL_INV operation is failing. So this is an issue with the fastreg stuff as I expected.
[^kprobes-off.sh,] [^kprobes.sh,] and trace1.txt contain the kprobes used and the tracing.
I'm trying to get more details now.
|
|
I should point our our upstream CI testing validates NFS/RDMA, iSer, SRP use of the fast reg feature and has been passing without issue.
|
|
I have attached refined kprobes and trace2.txt.
|
|
Here is the invalidate code:
int rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey)
{
<snip>
mr = rcu_dereference(
rkt->table[(rkey >> (32 - dev->dparms.lkey_table_size))]);
if (unlikely(!mr || mr->lkey != rkey || qp->ibqp.pd != mr->pd)) *** rkey is != mr->lkey
goto bail;
<snip>
bail:
rcu_read_unlock();
return -EINVAL;
}
Here is an excerpt from the trace:
It looks to me like Lustre is losing track of the keys for a particular MR?
|
|
Thanks, Mike and Dennis.
Serguei, please let us know when you (or someone) are working on this new information Cornelis came up with. Thank you.
|
|
It looks like Lustre is sending a gratuitous invalidate because of this code fragment:
/* There appears to be a bug in MLX5 code where you must
* invalidate the rkey of a new FastReg pool before first
* using it. Thus, I am marking the FRD invalid here. */
frd->frd_valid = false;
This is not wrong, but difference than other ULPs.
The following code is then executed before any fast reg has happened:
if (!frd->frd_valid) {
struct ib_rdma_wr *inv_wr;
__u32 key = is_rx ? mr->rkey : mr->lkey;
inv_wr = &frd->frd_inv_wr;
memset(inv_wr, 0, sizeof(*inv_wr));
inv_wr->wr.opcode = IB_WR_LOCAL_INV;
inv_wr->wr.wr_id = IBLND_WID_MR;
inv_wr->wr.ex.invalidate_rkey = key;
key = ib_inc_rkey(key);
*** updates keys in ib_mr, but not the rvt_mregion keys ***
ib_update_fast_reg_key(mr, key);
}
The following code uses struct rvt_mregion keys to validate and doesn't see the above key change in the ibmr and fails the invalidate. The rkey is correct, but the mr->lkey hasn't changed to match until the next
int rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey)
{
struct rvt_dev_info *dev = ib_to_rvt(qp->ibqp.device);
struct rvt_lkey_table *rkt = &dev->lkey_table;
struct rvt_mregion *mr;
if (rkey == 0)
return -EINVAL;
rcu_read_lock();
mr = rcu_dereference(
rkt->table[(rkey >> (32 - dev->dparms.lkey_table_size))]);
if (unlikely(!mr || mr->lkey != rkey || qp->ibqp.pd != mr->pd))
goto bail;
atomic_set(&mr->lkey_invalid, 1);
rcu_read_unlock();
return 0;
bail:
rcu_read_unlock();
return -EINVAL;
}
I'm working on the following fix:
|
|
Hi,
To me this looks like a very good candidate to fix the issue. Thanks for taking the time to look into this!
Thanks,
Serguei.
|
|
> To me this looks like a very good candidate to fix the issue.
Another potential fix is to delete the MLX5 hack. That should work as well.
|
|
The patch didn't work.
I need to do more analysis.
|
|
Hi Serguei,
Do you (or Amir, or anyone else you have easy access to) know if the MLX issue that prompted that hack has been fixed? The JIRA issue was LU-8752 and the commit was:
|
|
I reverted
and tested. Initially I had nonzero bandwidth, which is different than I recall seeing before. After a few seconds the bandwidth recorded went to 0. lctl dk shows:
|
|
I do know that patch proposed is just wrong. Testing against the struct rvt_mregion lkey should be correct.
I need to add another kprobe with the original kernel to look at the keys for all ib_alloc_mr() allocations from birth and track from that point to failures.
A portion of the comment says:
This fix just tags the MR as invalid on its creation if we are
using FastReg and that will force it to do an invalidate of the
rkey on first usage.
The inference is that the invalidate only happens on first MR use, but I don't see anywhere that sets frd_invalid to true? It looks like it will happen all the time for all MRs.
|
|
It is starting to look to me like there is a concurency issue where somehow the old key is subsequently passed to an invalidate.
Here is an invalidate for rkey 0x100:
Here is the one that fails:
Note the different CPU and task name.
|
|
Hi Mike and Serguei,
Do you have any update on this issue?
I'm expecting to get my OPA test cluster back again today, at which point I plan to more thoroughly compare the behavior of lnet_selftest both with and without the patch to invalidate the rkey before the mr is used.
thanks,
Olaf
|
|
There are two issues with the existing code and hfi1.
First there is what I suspect is a misplaced racy assignment:
void
kiblnd_fmr_pool_unmap(struct kib_fmr *fmr, int status)
{
<snip> This code returns the frd to the list
if (frd) {
frd->frd_valid = false;
spin_lock(&fps->fps_lock);
list_add_tail(&frd->frd_list, &fpo->fast_reg.fpo_pool_list);
spin_unlock(&fps->fps_lock);
fmr->fmr_frd = NULL; <- I think this should be before add
}
<snip> I suspect the NULL should be before adding to the list to avoid a race with the map
<snip> pulling the descriptor from the list
}
The mapping looks like:
int kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx,
struct kib_rdma_desc *rd, u32 nob, u64 iov,
struct kib_fmr *fmr)
{
<snip>This code dequeues the kib_fast_reg_descriptor from a list
struct kib_fast_reg_descriptor *frd;
#ifdef HAVE_IB_MAP_MR_SG
struct ib_reg_wr *wr;
int n;
#else
struct ib_rdma_wr *wr;
struct ib_fast_reg_page_list *frpl;
#endif
struct ib_mr *mr;
frd = list_first_entry(&fpo->fast_reg.fpo_pool_list,
struct kib_fast_reg_descriptor,
frd_list);
list_del(&frd->frd_list);
spin_unlock(&fps->fps_lock);
<snip> This code sets up the invalidate operation embbeded in the kib_fast_reg_descriptor
if (!frd->frd_valid) {
struct ib_rdma_wr *inv_wr;
__u32 key = is_rx ? mr->rkey : mr->lkey;
inv_wr = &frd->frd_inv_wr;
memset(inv_wr, 0, sizeof(*inv_wr));
inv_wr->wr.opcode = IB_WR_LOCAL_INV;
inv_wr->wr.wr_id = IBLND_WID_MR;
inv_wr->wr.ex.invalidate_rkey = key;
key = ib_inc_rkey(key);
ib_update_fast_reg_key(mr, key);
}
<snip> The code goes on to register the pages in ib_map_mr_sg and sets up the
<snip> frd_fastreg_wr embedded in the kib_fast_reg_descriptor
<snip>
<snip> The code then fuses the kib_fast_reg_descriptor kib_fmr in the kib_tx
fmr->fmr_key = is_rx ? mr->rkey : mr->lkey;
fmr->fmr_frd = frd; <--- here
fmr->fmr_pool = fpo;
return 0;
<snip> At this point no posts have been done and they are defered to
<snip> kiblnd_post_tx_locked().
}
The post send looks like:
static int
kiblnd_post_tx_locked(struct kib_conn *conn, struct kib_tx *tx, int credit)
__must_hold(&conn->ibc_lock)
{
<snip> This code sees frd from above and prepends WRs from the kib_fast_reg_descriptor
if (frd != NULL) {
if (!frd->frd_valid) {
wr = &frd->frd_inv_wr.wr;
wr->next = &frd->frd_fastreg_wr.wr;
} else {
wr = &frd->frd_fastreg_wr.wr;
}
frd->frd_fastreg_wr.wr.next = &tx->tx_wrq[0].wr;
will_post = true;
}
<snip> The post is here
if (lnet_send_error_simulation(tx->tx_lntmsg[0], &tx->tx_hstatus))
rc = -EINVAL;
else
#ifdef HAVE_IB_POST_SEND_RECV_CONST
rc = ib_post_send(conn->ibc_cmid->qp, wr,
(const struct ib_send_wr **)&bad);
#else
rc = ib_post_send(conn->ibc_cmid->qp, wr, &bad);
#endif
conn->ibc_last_send = ktime_get();
if (rc == 0)
return 0; <-- return here with the WRs from the mapping complete
<snip> At this point there is a tx with everthing ready to go, BUT
<snip> all posts using the tx until is unmapped will send the invalidate and fast reg
<snip> and the invalidate has the OLD key forever since nothing has been done to
<snip> remember and disable the invalid WRs for the next post using the tx.
}
Everytime the kib_tx is used after the first there will be a superflous OLD invalidate followed by an OLD fast req that will change the key to what it currently is.
For hfi1 the second invalidate will get a -EINVAL return code because the keys don't match.
There are two possible fixes:
- Add state to the kib_fast_reg_descriptor that keeps track of if the fast reg WRs have been posted and patch the post logic to only post the WRs if they had not already been posted.
- Make the rdmavt invalidate code allow the OLD invalidate and fast reg by only comparing the key bits above 7.
I'm about to attach the two patches for 1. The patch seems to fix the issue and the lnet_selftest works fine.
I'm getting ready to test the invalidate patch, but I should point out our current code works with SRP, iSer, and NFS RDMA as is.
|
|
linux-kernel-test.patch contains a potential upstream patch.
Older versions attached were incorrect.
|
|
linux-kernel-test.patch solves the issue, at the expense of extra processing whenever the kib_tx is reused.
|
|
Mike,
I tried the LND change you suggested on my setup using MOFED, it appears to be fine.
Please submit the LND patch for review, it will be easier to track with your ownership.
Thanks,
Serguei.
|
Please submit the LND patch for review, it will be easier to track with your ownership.
Which one? There are two patches. One is a bug fix in unmap. The other is the fix to insure duplicate WRs are not sent after the initial posts.
|
|
Mike
If you need help getting your gerrit account sorted out then it's best to email me directly rather than using JIRA 
Peter
|
|
Mike,
I was referring to the bug fix in the kiblnd_fmr_pool_unmap.
I have no way of testing the upstream patch with OPA. Is it possible that the kiblnd_fmr_pool_unmap fix is sufficient by itself?
Thanks,
Serguei.
|
I have no way of testing the upstream patch with OPA. Is it possible that the kiblnd_fmr_pool_unmap fix is sufficient by itself?
I tested the rdmavt upstream fix by rebuilding the 8.4 GA kernel from srpm adding the upstream patch. The unmap fix is not sufficient by itself. I determined that early on.
We either need an upstream patch to get pulled by Jason and point Red Hat to it, or use the second Lustre patch that avoids the superflous WRs that trigger the EINVAL.
It seems to me that a Lustre fix might be quicker?
Do we need a separate Jira for the map/unmap race fix?
|
|
I don't think we need a separate ticket for the map/unmap race fix. Peter can correct me if he disagrees.
|
|
Mike,
Thanks for tracking this down. Do you know why we saw this first with RHEL 8.4, but not RHEL 8.3?
|
Do you know why we saw this first with RHEL 8.4, but not RHEL 8.3?
Upstream removed FMR as of 5.8 and it looks like the 8.4 RDMA code took that in.
8.3 still has the FMR and it appears that this code will prefer it:
#ifdef HAVE_FMR_POOL_API
if (dev->ibd_dev_caps & IBLND_DEV_CAPS_FMR_ENABLED)
rc = kiblnd_alloc_fmr_pool(fps, fpo);
else
#endif
rc = kiblnd_alloc_freg_pool(fps, fpo, dev->ibd_dev_caps);
if (rc)
goto out_fpo;
fpo->fpo_deadline = ktime_get_seconds() + IBLND_POOL_DEADLINE;
fpo->fpo_owner = fps;
*pp_fpo = fpo;
return 0;
I don't see any override when both are available?
|
|
Here are the patches for Lustre:
- 01-move_null.patch
- 02-post_state.patch
|
|
Mike Marciniszyn (mike.marciniszyn@cornelisnetworks.com) uploaded a new patch: https://review.whamcloud.com/44189
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 071c8c7faa322be1a42d424ff0d83c3113c80140
|
|
Mike Marciniszyn (mike.marciniszyn@cornelisnetworks.com) uploaded a new patch: https://review.whamcloud.com/44190
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 778962a825eeee5b4754664a6baabf61e982aa38
|
|
Currently I have unit tested both bulk read and write with opa cards and RHEL8.4.
I'm trying to find an MLX card to test with that as well.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/44189/
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 023113fb8946f3565529e7327fdcd90ab9db3ba3
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/44190/
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5930576791e864529e6ef9b46f3e09cc4b635fc2
|
|
I saw those two Lustre patches were merged to master. Has someone been able to test them on MLX to confirm they don't cause new issues there? Thanks
|
|
Before Mike pushed them, I tried these patches on my local setup that uses LTS MOFED 4.9 and cx-2 cards. Ran lnet_selftest, didn't see any issues.
|
Before Mike pushed them, I tried these patches on my local setup that uses LTS MOFED 4.9 and cx-2 cards. Ran lnet_selftest, didn't see any issues.
I will also test on some MLX cards. I will try the stock 8.4 kernel vs. MOFED though.
|
|
Great, thank you both.
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/44216
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: d33a2dcf34fcea3adaf46d1a806405a9c334adc0
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/44217
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 8c70059776f2bdc7a68a27e35bb5dc36763d3dd6
|
05:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
I was able to run lnet_selftest on a vanilla RH 8.4 install:
The only issue I ran into was a panic trying to reboot. The servers required a powercyle.
|
|
Gian-Carlo DeFazio (defazio1@llnl.gov) uploaded a new patch: https://review.whamcloud.com/44295
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_14
Current Patch Set: 1
Commit: 8db93a26f7e99f140ccd4a0fd3e35f4e9f71b8ec
|
|
Gian-Carlo DeFazio (defazio1@llnl.gov) uploaded a new patch: https://review.whamcloud.com/44296
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_14
Current Patch Set: 1
Commit: 1105bfbfad5002bc673ff568c0720bcebc5d095a
|
|
Mike, Serguei, this works on our test system
|
|
Landed for 2.15
|
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/44216/
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 173d60a3274d19bc1d9811b6e1b09aac2b25f221
|
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/44217/
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 96d7dcf4e773e6026a590e4596ef30ac8a4a5061
|
|
"Andreas Dilger <adilger@whamcloud.com>" merged in patch https://review.whamcloud.com/44295/
Subject: LU-14733 o2iblnd: Move racy NULL assignment
Project: fs/lustre-release
Branch: b2_14
Current Patch Set:
Commit: 380be07fcca1f76564d1f29e58f2d8d5f8f530c8
|
|
"Andreas Dilger <adilger@whamcloud.com>" merged in patch https://review.whamcloud.com/44296/
Subject: LU-14733 o2iblnd: Avoid double posting invalidate
Project: fs/lustre-release
Branch: b2_14
Current Patch Set:
Commit: 29da7cba3e7b3461d895010c7f7284b9649aba52
|
Generated at Sat Feb 10 03:12:18 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
01-move_null.patch
kprobes-off.sh