https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6723 Lustre <p>While testing setting map_on_demand with the patch from <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> I discovered set map_on_demand to any value on our Cray routers prevented lustre from functioning. This looks like a bug in the o2iblnd driver which only shows up on our Cray nodes.</p> Cray routers running SLES11 SP3. Found this issue exist for all lustre versions. LU-6723

Setting map_on_demand for o2iblnd driver prevents lustre bring up.

Bug Minor Resolved Won't Fix Amir Shehata James A Simmons lnet Mon, 15 Jun 2015 18:59:29 +0000 Wed, 16 Dec 2015 20:45:17 +0000 Wed, 16 Dec 2015 20:45:17 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.5.4 0 8 <p>As a note this happened also when the patch from <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> is not applied.</p> <p>James, could you please provide a bit more information about what you mean by "Cray routers prevented Lustre from functioning"? Any errors in the logs? Does "lctl ping" work? Does the IB-level network testing still work?</p> <p>Is Cray running a customized OFED? It may be that this isn't a Lustre/LNet problem at all.</p> <p>Is is the errors that appear on the OSS nodes when I enabled map_on_demand on the Cray routers. </p> <p>00000020:02000400:10.0:1433178825.928974:0:28309:0:(tgt_handler.c:1834:tgt_brw_read()) sultan-OST0034: Bulk IO read error with b9cf5051-0ff9-6cf9-cd67-9364a2516176 (at 30@gni1), client will retry: rc -110<br/> 00000020:00000001:10.0:1433178825.947041:0:28309:0:(tgt_handler.c:1851:tgt_brw_read()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92)<br/> 00010000:00000080:10.0:1433178825.947043:0:28309:0:(ldlm_lib.c:2427:target_committed_to_req()) @@@ not sending l</p> <p>The Cray routers are using the mlx5 driver from the OFED 3.12 stack. Realizing what the problem is I need to collect logs from the routers so we know what is really going on. The OSS bulk timeouts are a symptom of the real problem.</p> <p>As a small note the OSS that also had problems when map_on_demand is enabled was running RHEL6.5 with the default distro infiniband stack. So it is not a inifinband issue.</p> <p>Hi James,<br/> Does the issue in this ticket still exist?</p> <p>I haven't tried in a while. Will do.</p> <p>Just tried it. Now that the OFED stack has been updated to a newer 3.12 the mlx5 driver no longer supports FMR so this issue has gone away. I will be trying the <a href="https://jira.whamcloud.com/browse/LU-5783" title="o2iblnd: investigate new memory registration mechanisms" class="issue-link" data-issue-key="LU-5783"><del>LU-5783</del></a> work very soon on our Cray routers to see if I hit memory issues. In that case the bugs can be reported under that ticket. You can close this ticket.</p> <p>Thank you James.</p> Duplicate LU-6748 Related LU-6748 Development Rank 1|hzxfsf: Rank (Obsolete) 9223372036854775807 Severity