https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7101 Lustre <p>MLX5 does not support FMR. With the removal of PMR (<a href="https://jira.whamcloud.com/browse/LU-6850" title="Remove use of ib_reg_phys_mr() from o2iblnd " class="issue-link" data-issue-key="LU-6850"><del>LU-6850</del></a>) we need a way to support MLX5. </p> <p>To do so the following solution is devised. FMR is enabled by setting map-on-demand to: 0 &lt; value &lt;=256. </p> <p>This represents a problem for coexistence nodes with both OPA and MLX5. OPA performance is greatly enhanced by setting map-on-demand, however if map-on-demand is set for MLX5, then it will not work.</p> <p>We need to be able to support per-NI map-on-demand value; therefore when OPA net is configured it can use optimal map-on-demand value, but when MLX5 is configured map-on-demand can be set to 0, disabling it.</p> <p>However, this raises another issue which is different map-on-demand values across fabrics. LNet currently doesn't support this. However <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> adds support for this scenario. This problem is relevant for both OPA support and for support of clusters which have both MLX5 and MLX4 interconnected.</p> <p>The proposed solution is consistent of three patches.<br/> 1. <a href="https://jira.whamcloud.com/browse/LU-6850" title="Remove use of ib_reg_phys_mr() from o2iblnd " class="issue-link" data-issue-key="LU-6850"><del>LU-6850</del></a> patch which removed support for PMR<br/> 2. <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> support for different map-on-demand and peertxcredits values across the network.<br/> 3. The patch for this LU, which adds support for per NI map-on-demand.</p> <p>Future patch will add support for dynamic setting of map-on-demand, but that's a future feature not required to address the immediate need.</p> LU-7101

Lnet: Support per NI map-on-demand

Bug Major Resolved Fixed Amir Shehata Amir Shehata Fri, 4 Sep 2015 00:47:36 +0000 Mon, 30 Jan 2017 16:25:02 +0000 Mon, 11 Apr 2016 13:35:06 +0000 Lustre 2.9.0 0 16 <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16367" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16367</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7101" title="Lnet: Support per NI map-on-demand" class="issue-link" data-issue-key="LU-7101"><del>LU-7101</del></a> lnet: per NI map-on-demand value<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e77ed3e90197a749ba4d301ae5414731f861b5cd</p> <p>I just tested this patch set and got it to work. For this set of test I didn't use map_on_demand at all. For the mlx5 driver map_on_demand will not work. It always gives the follow error no matter what value I set to map_on_demand:</p> <p>LNetError: 8048:0:(o2iblnd.c:2242:kiblnd_net_init_pools()) Can't set fmr pool size (512) &lt; ntx / 4(1280)<br/> [ 3766.868871] LNetError: 8048:0:(o2iblnd.c:3110:kiblnd_startup()) Failed to initialize NI pools: -22</p> <p>My config string is:</p> <p>options ko2iblnd timeout=100 credits=2560 ntx=5120 peer_credits=63 concurrent_sends=63</p> <p>What I did find that work with the mlx5 driver is:</p> <p>options ko2iblnd timeout=100 credits=2560 ntx=5120 peer_credits=16 concurrent_sends=16</p> <p>On the server side I'm still using the config string: </p> <p>options ko2iblnd timeout=100 credits=2560 ntx=5120 peer_credits=63 concurrent_sends=63</p> <p>I haven't tried map_on_demand on the server side yet. Any suggestions to bump up the peer_credits?</p> <p>In lustre's current o2iblnd LND map_on_demand != 0 is equivalent to enabling FMR. Since mlx5 doesn't support FMR it will fail. The error you are seeing is because fmr_pool_size defaults to 512 but even if you change it to a larger value though it should then fail in kiblnd_create_fmr_pool on the call to ib_create_fmr_pool.</p> <p>Latest patch update has a new look for lnetctl net show -v</p> <p>net:</p> <ul class="alternate" type="square"> <li>net: lo<br/> nid: 0@lo<br/> status: up<br/> tunables:<br/> peer_timeout: 0<br/> peer_credits: 0<br/> peer_buffer_credits: 0<br/> credits: 0<br/> CPT: "<span class="error">&#91;0,0&#93;</span>"</li> <li>net: o2ib1<br/> nid: 10.37.248.19@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> tunables:<br/> peer_timeout: 180<br/> peer_buffer_credits: 0<br/> peer_credits: 63<br/> credits: 2560<br/> CPT: "<span class="error">&#91;0,0&#93;</span>"<br/> LND tunables:<br/> use_privileged_port: 1<br/> require_privileged_port: 0<br/> dev_failover: 0<br/> fmr_cache: 1<br/> fmr_flush_trigger: 1024<br/> fmr_pool_size: 1280<br/> map_on_demand: 256<br/> concurrent_sends: 63<br/> ib_mtu: 0<br/> keepalive: 100<br/> rnr_retry_count: 6<br/> retry_count: 5<br/> ipif_name: ib0<br/> peer_credits_hiw: 31<br/> ntx: 5120<br/> nscheds: 0<br/> timeout: 100<br/> cksum: 0<br/> service: 987</li> </ul> <p>Updated the patch to support setting FMR pool parameters as well. The patch is flexible enough to allow different settings on different IB ports on the same node. See the output of lnetctl net show -v<br/> This is big step forward in that we can now support different types of IIB hardware in the same node and configure each one independently.</p> <p>net:</p> <ul class="alternate" type="square"> <li>net: lo<br/> nid: 0.0.0.0@lo<br/> status: up<br/> tunables:<br/> peer_timeout: 0<br/> peer_credits: 0<br/> peer_buffer_credits: 0<br/> credits: 0</li> <li>net: o2ib1<br/> nid: pike11-ib0@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> tunables:<br/> peer_timeout: 100<br/> peer_credits: 16<br/> peer_buffer_credits: 0<br/> credits: 2560<br/> CPT: "<span class="error">&#91;0,0&#93;</span>"<br/> LND tunables:<br/> peercredits_hiw: 8<br/> map_on_demand: 256<br/> concurrent_sends: 32<br/> fmr_pool_size: 1280<br/> fmr_flush_trigger: 1024<br/> fmr_cache: 1</li> <li>net: o2ib2<br/> nid: pike12-ib1@o2ib2<br/> status: up<br/> interfaces:<br/> 0: ib1<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 16<br/> peer_buffer_credits: 0<br/> credits: 2560<br/> CPT: "<span class="error">&#91;0,0,0,0&#93;</span>"<br/> LND tunables:<br/> peercredits_hiw: 8<br/> map_on_demand: 0<br/> concurrent_sends: 16<br/> fmr_pool_size: 512<br/> fmr_flush_trigger: 384<br/> fmr_cache: 1</li> </ul> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16367/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16367/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7101" title="Lnet: Support per NI map-on-demand" class="issue-link" data-issue-key="LU-7101"><del>LU-7101</del></a> lnet: per NI map-on-demand value<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 94d1ce46562bd36c8959a9458cacabb7f6df681f</p> <p>Landed to master for 2.9.0</p> Duplicate Related LU-8022 LU-3322 Development Rank 1|hzxmhb: Rank (Obsolete) 9223372036854775807 Severity