https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5828 Lustre <p>Cannot verify if the routing buffer has been set or not, the same happens on small and large buffers as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-27 proc]# more /sys/module/lnet/parameters/forwarding enabled [root@onyx-27 proc]# lnetctl set tiny_buffers 2048 [root@onyx-27 proc]# lnetctl routing show routing: - cpt[0]: tiny: npages: 0 nbuffers: 512 credits: 512 mincredits: 512 small: npages: 1 nbuffers: 4096 credits: 4096 mincredits: 4096 large: npages: 256 nbuffers: 256 credits: 256 mincredits: 256 - enable: 1 </pre> </div></div> server and client: lustre-master build #2702 LU-5828

DLC: Cannot verify if the routing buffer has been set or not

Bug Minor Resolved Fixed Amir Shehata Sarah Liu Thu, 30 Oct 2014 21:49:33 +0000 Fri, 20 Feb 2015 15:13:00 +0000 Fri, 20 Feb 2015 15:13:00 +0000 Lustre 2.7.0 Lustre 2.7.0 0 6 <p>When I tested on my VM it works. Please try the following steps:</p> <p>1. Unload lnet<br/> 2. reload lnet<br/> 3. enable routing via "lnetctl set routing 1"<br/> 4. check values via "lnetctl routing show"<br/> 5. increase the value of one of the buffers via "lnetctl set tiny_buffers &lt;number&gt;"<br/> 6. check values via "lnetctl routing show"<br/> 7. Please paste the output of the above commands in this bug.</p> <p>Also note that if you reduce the number of buffers the reduction doesn't show immediately, as buffers are freed when they have been used and are being returned. So as traffic starts up, you'll see that the buffers are reduced.</p> <p>The best way to test so you can immediately see, is to increase the number of buffers.</p> <p>Hi this is what I got</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-27 ~]# lsmod|grep lnet [root@onyx-27 ~]# modprobe lnet LNet: HW CPU cores: 32, npartitions: 4 alg: No test for adler32 (adler32-zlib) alg: No test for crc32 (crc32-table) alg: No test for crc32 (crc32-pclmul) padlock: VIA PadLock Hash Engine not detected. [root@onyx-27 ~]# lsmod|grep lnet lnet 343308 0 libcfs 491216 1 lnet [root@onyx-27 ~]# lnetctl set routing 1 add: - routing: errno: -100 descr: "cannot enable routing Network is down" [root@onyx-27 ~]# lctl network up LNet: Added LNI 192.168.4.65@o2ib [8/256/0/180] LNet: Added LNI 10.2.4.65@tcp [8/256/0/180] LNet: Accept secure, port 7988 LNET configured [root@onyx-27 ~]# lnetctl set routing 1 [root@onyx-27 ~]# lnetctl routing show routing: - cpt[0]: tiny: npages: 0 nbuffers: 512 credits: 512 mincredits: 512 small: npages: 1 nbuffers: 4096 credits: 4096 mincredits: 4096 large: npages: 256 nbuffers: 256 credits: 256 mincredits: 256 - enable: 1 [root@onyx-27 ~]# lnetctl set tiny_buffers 1024 [root@onyx-27 ~]# lnetctl routing show routing: - cpt[0]: tiny: npages: 0 nbuffers: 512 credits: 512 mincredits: 512 small: npages: 1 nbuffers: 4096 credits: 4096 mincredits: 4096 large: npages: 256 nbuffers: 256 credits: 256 mincredits: 256 - enable: 1 [root@onyx-27 ~]# </pre> </div></div> <p>can you look at /var/log/messages to see if there are any errors?</p> <p>onyx-27 has 4 CPTs. However due to overloading of iterator variable in the show routing information function, only the first CPT was displayed.</p> <p><a href="http://review.whamcloud.com/12593" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12593</a></p> <p>Just note however, the number of buffers specified on the lnetctl command line is the total number of buffers for all CPTs. That value is then divided by the number of CPTs, so that each CPT has an equal number of buffers. The per CPT number of buffers need to be greater than the minimum per buffer type:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#define LNET_NRB_TINY_MIN»······512»····<span class="code-comment">/* min value <span class="code-keyword">for</span> each CPT */</span> #define LNET_NRB_SMALL_MIN»·····4096»···<span class="code-comment">/* min value <span class="code-keyword">for</span> each CPT */</span> #define LNET_NRB_LARGE_MIN»·····256»····<span class="code-comment">/* min value <span class="code-keyword">for</span> each CPT */</span> </pre> </div></div> <p>The logic is that if you try to set the per CPT number of buffers below the minimum, it's not accepted silently.</p> <p>As an example the minimum per CPT for tiny buffers is 512<br/> If you have 4 CPTs and you try to set the value of the total number of tiny buffers to 1024. 1024/4 = 256 &lt; 512 so you don't see the change.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12593/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12593/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5828" title="DLC: Cannot verify if the routing buffer has been set or not" class="issue-link" data-issue-key="LU-5828"><del>LU-5828</del></a> lnet: showing buffers problem with mulitple CPTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b4b00be8a93cf06f232d3edc613f03d06b112d32</p> <p>This has been verified in build #2808</p> <p>Reopening to add fix version.</p> Development Rank 1|hzwzv3: Rank (Obsolete) 16349 Severity