[LU-8752] mlx5_warn:mlx5_0:dump_cqe:257:

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0
• Environment: mlnx ofed3.2; lustre-2.7.2-2nas-fe; Linux elrtr1 3.0.101-77.1.20160630-nasa #1 SMP Thu Jun 30 00:56:32 UTC 2016 (a082ea6) x86_64 x86_64 x86_64 GNU/Linux
• Severity: 2
• Assignee: Doug Oucharek (Inactive)
• Reporter: Mahmoud Hanafi

Description

Running lnet selftest on a mlx5 card we get these errors.

[1477328975.069684] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10912): dump error cqe
[1477328975.085684] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10906): dump error cqe
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 08007806 2500002f 00085dd0
[1477328975.085684] LustreError: 11028:0:(brw_test.c:388:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-10.151.27.25@o2ib: -5
[1477328975.085684] LustreError: 11028:0:(brw_test.c:362:brw_server_rpc_done()) Bulk transfer to 12345-10.151.27.25@o2ib has failed: -5
[1477328975.093683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10922): dump error cqe
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 08007806 25000030 000842d0
[1477328975.105683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10915): dump error cqe
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 08007806 25000031 000843d0
[1477328975.113683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10900): dump error cqe
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 08007806 25000032 000840d0
[1477328975.121683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10900): dump error cqe
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 08007806 25000033 000841d0
[1477328975.129683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10915): dump error cqe
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 08007806 2500002e 00085cd0
[1477328975.133683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10907): dump error cqe
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 08007806 25000034 000846d0
[1477328975.205682] 00000000 00000000 00000000 00000000
[1477328975.281682] 00000000 00000000 00000000 00000000
[1477328975.305681] 00000000 00000000 00000000 00000000
[1477328975.305681] 00000000 08007806 2500002d 000b57d0

Issue Links

• is duplicated by: LU-8693 ko2iblnd recieving IB_WC_MW_BIND_ERR errors.

Comments

[Mahmoud Hanafi]

Here is some feed back from MLX.

"

According to IB spec state should be set to Free.

Under "MEMORY REGION TYPES" section there is an explanation of states, specifically:
Table: "Memory Region States Summary"
"The following table summarizes the states of Memory Regions L_Keys and R_Keys and the operations allowed on each state:"
Looking at Property / Operation Allowed: for "Fast Register", there are 3 possible states: Invalid, Free, Valid.
The only allowed state is - Free.

"

[Doug Oucharek (Inactive)]

The 2.7FE patch is: https://review.whamcloud.com/24336

[Nathan Dauchy (Inactive)]

Yes, hopefully it is trivial, but we will want to use the 2.7fe patch. Thanks!

[Doug Oucharek (Inactive)]

OmniPath is not affected by this patch as it uses FMR and not FastReg.  So this change should only affect MLX5 based cards.

Did you need me to push a 2.7FE patch?

[Mahmoud Hanafi]

Thanks we will build and test it.

 

[Doug Oucharek (Inactive)]

I have submitted a fix to this issue above (ignore the earlier Gerrit patch as it was just for my debugging).  I have validated it on our mlx4 / mlx5 mixed cluster.  I still need to validate it on OmniPath to ensure it does not cause a problem there.

[Gerrit Updater]

Doug Oucharek ([e-mail]) uploaded a new patch: https://review.whamcloud.com/24306
Subject: LU-8752 lnet: Stop MLX5 triggering a dump_cqe
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3dcf679c7f1e2aa99d99a10deb65ac2faa30a629

[Gerrit Updater]

Doug Oucharek ([e-mail]) uploaded a new patch: https://review.whamcloud.com/24162
Subject: LU-8752 lnet: Debugging Patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 235b2ce7d07ae80e31a170cb7e424f859dce97b1

[Doug Oucharek (Inactive)]

Thank you for that feedback.  I will dig into the code for the path where we are making this mistake.

[Mahmoud Hanafi]

Here is update from mellanox engineering

"The failure is happening because of fast reg mr called twice, second time the mkey is not free - but it set for a check if free and its meaning the operation will succeed only if mkey is free.

In order to call for second fast reg mr customer needed to do local invalidate before that call.

(These operation are explained in IB spec)."

[Doug Oucharek (Inactive)]

Another interesting data point:

I flipped the from and to parameters in your test making it "--from mlx5_host --to mlx4_host". In theory, the issue should happen when I then do a write as this is almost the same thing as what you have. However, in this case, I am not able to reproduce the problem with either read or write.

I believe the difference when you flip the from and to parameters is who initiates the test.