[LU-8752] mlx5_warn:mlx5_0:dump_cqe:257:

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.7.0
• Environment: mlnx ofed3.2; lustre-2.7.2-2nas-fe; Linux elrtr1 3.0.101-77.1.20160630-nasa #1 SMP Thu Jun 30 00:56:32 UTC 2016 (a082ea6) x86_64 x86_64 x86_64 GNU/Linux
• Severity: 2
• Assignee: Doug Oucharek (Inactive)
• Reporter: Mahmoud Hanafi

Description

Running lnet selftest on a mlx5 card we get these errors.

[1477328975.069684] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10912): dump error cqe
[1477328975.085684] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10906): dump error cqe
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 00000000 00000000 00000000
[1477328975.085684] 00000000 08007806 2500002f 00085dd0
[1477328975.085684] LustreError: 11028:0:(brw_test.c:388:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-10.151.27.25@o2ib: -5
[1477328975.085684] LustreError: 11028:0:(brw_test.c:362:brw_server_rpc_done()) Bulk transfer to 12345-10.151.27.25@o2ib has failed: -5
[1477328975.093683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10922): dump error cqe
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 00000000 00000000 00000000
[1477328975.093683] 00000000 08007806 25000030 000842d0
[1477328975.105683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10915): dump error cqe
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 00000000 00000000 00000000
[1477328975.105683] 00000000 08007806 25000031 000843d0
[1477328975.113683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10900): dump error cqe
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 00000000 00000000 00000000
[1477328975.113683] 00000000 08007806 25000032 000840d0
[1477328975.121683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10900): dump error cqe
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 00000000 00000000 00000000
[1477328975.121683] 00000000 08007806 25000033 000841d0
[1477328975.129683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10915): dump error cqe
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 00000000 00000000 00000000
[1477328975.129683] 00000000 08007806 2500002e 00085cd0
[1477328975.133683] mlx5_warn:mlx5_0:dump_cqe:257:(pid 10907): dump error cqe
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 00000000 00000000 00000000
[1477328975.133683] 00000000 08007806 25000034 000846d0
[1477328975.205682] 00000000 00000000 00000000 00000000
[1477328975.281682] 00000000 00000000 00000000 00000000
[1477328975.305681] 00000000 00000000 00000000 00000000
[1477328975.305681] 00000000 08007806 2500002d 000b57d0

Issue Links

• is duplicated by: LU-8693 ko2iblnd recieving IB_WC_MW_BIND_ERR errors.

Comments

[Jay Lan (Inactive)]

Hi Peter,
NASA Ames is running 2.7.2+ in production, but we started testing 2.9.0 client on SLES12 SP2.

• jay

[Jian Yu]

Hi James,
Here is the patch for FE 2.8.x release: https://review.whamcloud.com/24365

[James A Simmons]

ORNL needs it for the 2.8 release.

[Peter Jones]

Jay

Are you running with 2.9.x? I had thought that you were using 2.7.x...

Peter

[Jay Lan (Inactive)]

Do we need this patch for Lustre 2.9 release?

[Mahmoud Hanafi]

Here is some feed back from MLX.

"

According to IB spec state should be set to Free.

Under "MEMORY REGION TYPES" section there is an explanation of states, specifically:
Table: "Memory Region States Summary"
"The following table summarizes the states of Memory Regions L_Keys and R_Keys and the operations allowed on each state:"
Looking at Property / Operation Allowed: for "Fast Register", there are 3 possible states: Invalid, Free, Valid.
The only allowed state is - Free.

"

[Doug Oucharek (Inactive)]

The 2.7FE patch is: https://review.whamcloud.com/24336

[Nathan Dauchy (Inactive)]

Yes, hopefully it is trivial, but we will want to use the 2.7fe patch. Thanks!

[Doug Oucharek (Inactive)]

OmniPath is not affected by this patch as it uses FMR and not FastReg.  So this change should only affect MLX5 based cards.

Did you need me to push a 2.7FE patch?

[Mahmoud Hanafi]

Thanks we will build and test it.

 

[Doug Oucharek (Inactive)]

I have submitted a fix to this issue above (ignore the earlier Gerrit patch as it was just for my debugging).  I have validated it on our mlx4 / mlx5 mixed cluster.  I still need to validate it on OmniPath to ensure it does not cause a problem there.