[LU-6723] Setting map_on_demand for o2iblnd driver prevents lustre bring up.

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0, Lustre 2.8.0, Lustre 2.5.4
• Type: Bug
• Priority: Minor
• Reporter: James A Simmons
• Assignee: Amir Shehata (Inactive)
• Resolution: Won't Fix
• Labels: lnet

Environment

Cray routers running SLES11 SP3. Found this issue exist for all lustre versions.

• Severity: 3

Description

While testing setting map_on_demand with the patch from LU-3322 I discovered set map_on_demand to any value on our Cray routers prevented lustre from functioning. This looks like a bug in the o2iblnd driver which only shows up on our Cray nodes.

Issue Links

• is duplicated by: LU-6748 excessive client reconnect to OSS ser...
• is related to: LU-6748 excessive client reconnect to OSS ser...

Comments

[James A Simmons]

As a note this happened also when the patch from LU-3322 is not applied.

[Andreas Dilger]

James, could you please provide a bit more information about what you mean by "Cray routers prevented Lustre from functioning"? Any errors in the logs? Does "lctl ping" work? Does the IB-level network testing still work?

Is Cray running a customized OFED? It may be that this isn't a Lustre/LNet problem at all.

[James A Simmons]

Is is the errors that appear on the OSS nodes when I enabled map_on_demand on the Cray routers.

00000020:02000400:10.0:1433178825.928974:0:28309:0:(tgt_handler.c:1834:tgt_brw_read()) sultan-OST0034: Bulk IO read error with b9cf5051-0ff9-6cf9-cd67-9364a2516176 (at 30@gni1), client will retry: rc -110
00000020:00000001:10.0:1433178825.947041:0:28309:0:(tgt_handler.c:1851:tgt_brw_read()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92)
00010000:00000080:10.0:1433178825.947043:0:28309:0:(ldlm_lib.c:2427:target_committed_to_req()) @@@ not sending l

The Cray routers are using the mlx5 driver from the OFED 3.12 stack. Realizing what the problem is I need to collect logs from the routers so we know what is really going on. The OSS bulk timeouts are a symptom of the real problem.

[James A Simmons]

As a small note the OSS that also had problems when map_on_demand is enabled was running RHEL6.5 with the default distro infiniband stack. So it is not a inifinband issue.

[Jian Yu]

Hi James,
Does the issue in this ticket still exist?

[James A Simmons]

I haven't tried in a while. Will do.

[James A Simmons]

Just tried it. Now that the OFED stack has been updated to a newer 3.12 the mlx5 driver no longer supports FMR so this issue has gone away. I will be trying the LU-5783 work very soon on our Cray routers to see if I hit memory issues. In that case the bugs can be reported under that ticket. You can close this ticket.

[Jian Yu]

Thank you James.