https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8507 Lustre <p>The code to allow peer_credits to be set per NI was originally "left inactive" because there were concerns about peer_credits interfering with the ability for IB nodes to connect to each other when peer_credits are not the same (peer_credits controls the queue depth for IB).</p> <p>With <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>, the values do not have to match so it is now safe to enable this code so peer_credits can be set per NI.</p> LU-8507

DLC: Enable peer_credit management

Bug Major Resolved Fixed Doug Oucharek Doug Oucharek Tue, 16 Aug 2016 16:51:54 +0000 Fri, 4 Aug 2017 18:55:21 +0000 Fri, 2 Sep 2016 04:44:01 +0000 Lustre 2.9.0 Lustre 2.9.0 0 5 <p>Doug Oucharek (doug.s.oucharek@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21948" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21948</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8507" title="DLC: Enable peer_credit management" class="issue-link" data-issue-key="LU-8507"><del>LU-8507</del></a> lnet: Enable setting per NI peer_credits<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 54959518d88190ebccf3a1737344c6579a998842</p> <p>Will this be meant mostly for back porting since multi-rail fixes this.</p> <p>I don't think multi-rail will fix this. Customers may want to adjust the peer_credit value per NI using DLC even with multi-rail.</p> <p>With this patch I will no longer need to have a ko2ilnd modprobe config file which is awesome. So I tried your patch but I'm seeing some changes in what values are being set. With a modprobe config file with peer_credits=63 I see the following with lnetctl export &gt; settings.conf.<br/> This is the desired settings and what I placed in my setting.conf for lnetctl import.</p> <p>BTW the command I used to setup my configure for both cases is:</p> <p>modprobe lnet;lnetctl lnet configure;lnetctl import &lt; ~/settings.conf</p> <ul class="alternate" type="square"> <li>net: o2ib1<br/> nid: 10.37.202.11@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> lnd tunables:<br/> peercredits_hiw: 62<br/> map_on_demand: 16<br/> concurrent_sends: 31<br/> fmr_pool_size: 1280<br/> fmr_flush_trigger: 1024<br/> fmr_cache: 1<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 63<br/> peer_buffer_credits: 0<br/> credits: 2560<br/> CPT: "<span class="error">&#91;0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&#93;</span>"</li> </ul> <p>Here is the default settings when I remove my modprobe conf file with lnetctl export &gt; setttings.conf</p> <ul class="alternate" type="square"> <li>net: o2ib1<br/> nid: 10.37.202.11@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> lnd tunables:<br/> peercredits_hiw: 7<br/> map_on_demand: 16<br/> concurrent_sends: 16<br/> fmr_pool_size: 1280<br/> fmr_flush_trigger: 1024<br/> fmr_cache: 1</li> </ul> <p>As you can see peercredits_hiw and concurrent_sends is set to what I want. Any ideas why?</p> <p>Dough I figured out the problem. Its a old bug so this need to be back ported to several branches. Basically the data the user passed in using<br/> struct lnet_ioctl_config_data wasn't being used to set struct lnet_ni fields like ni-&gt;ni_peertxcredits until after lnd-&gt;lnd_startup() was called. This meant the o2iblnd driver was never seeing the ni-&gt;ni_peerXXX fields changed by the lnet_ioctl_config_data. So I moved the setting of the ni-&gt;ni_peerXXX fields with the lnet_ioctl_config_data before the call to lnd-&gt;lnd_startup(). Give it a try now.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21948/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21948/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8507" title="DLC: Enable peer_credit management" class="issue-link" data-issue-key="LU-8507"><del>LU-8507</del></a> lnet: Enable setting per NI peer_credits<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ebac11d264b306b694684f1ee91080a5e31684d4</p> <p>Landed for 2.9</p> Duplicate Related Development Rank 1|hzykvj: Rank (Obsolete) 9223372036854775807 Severity