https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2950 Lustre <p>Port a change made by Fujitsu where they added a new lnet module param: routes_config, which can specify a file that contains a set of routes to be used by lnet</p> LU-2950

Port LNET routes_config change done by Fujitsu in latest lustre

Improvement Major Resolved Fixed Amir Shehata Amir Shehata Fujitsu lnet Tue, 12 Mar 2013 18:17:36 +0000 Thu, 7 Aug 2014 20:37:01 +0000 Thu, 1 May 2014 16:14:24 +0000 Lustre 2.5.0 Lustre 2.4.0 Wed, 20 Mar 2013 00:00:00 +0000 0 10 <p>Please be aware of: <a href="https://jira.whamcloud.com/browse/LU-1071" title="able to handle many lnet routes" class="issue-link" data-issue-key="LU-1071"><del>LU-1071</del></a> able to handle many lnet routes</p> <p>I really don't like the fact of reading a file into the kernel space but I guess it is seen as a necessary evil at this point. Please safeguard all file actions as much as you can.</p> <p>We took Issac's suggestion and we created a shell script that reads a routes config file then uses lctl to configure lnet with the routes. This avoids opening files in kernel space.</p> <p>test files</p> <p>We created two scripts. One to convert from the legacy syntax to the new syntax, and the second to take a file of the new syntax and configure lustre using lctl.</p> <p>The legacy syntax (which is currently being used in the lustre.conf file):<br/> &lt;network&gt; <span class="error">&#91;&lt;hop&gt;&#93;</span> &lt;gateway&gt;@&lt;exit network&gt;<span class="error">&#91;:&lt;priority&gt;&#93;</span></p> <p>The new syntax (which is used in the new scripts):<br/> &lt;network&gt;: </p> { gateway: &lt;gateway&gt;@&lt;exit network&gt; [hope: &lt;hop&gt;] [priority: &lt;prioirty&gt;] } <p>I would think the intention is to move the syntax to the new form in the future, so there isn't three different syntax for configuring routes.</p> <p>Below is sample test runs. I have also attached two files to this Bug that were used for testing the scripts.</p> <p><span class="error">&#91;ashehata@localhost scripts&#93;</span>$ ~/lustre-master/lustre/scripts/lustre_routes_conversion legacy_routes_test new_routes_test<br/> tcp1 23 192.168.213.1@tcp:1 tcp5 34 193.30.4.3@tcp:4<br/> tcp2 54 10.1.3.2@tcp<br/> tcp3 10.3.4.3@tcp:3<br/> tcp4 10.3.3.4@tcp<br/> lustre_routes_conversion: converted routes written to test9<br/> <span class="error">&#91;ashehata@localhost scripts&#93;</span>$ ~/lustre-master/lustre/scripts/lustre_routes_config --dry-run --verbose new_routes_test<br/> lctl --net tcp1 add_route 192.168.213.1@tcp 23 1<br/> lctl --net tcp5 add_route 193.30.4.3@tcp 34 4<br/> lctl --net tcp2 add_route 10.1.3.2@tcp 54 4<br/> lctl --net tcp3 add_route 10.3.4.3@tcp 1 3<br/> lctl --net tcp4 add_route 10.3.3.4@tcp 1 3</p> <p>Can you submit your scripts into gerrit? </p> <p>The code inspection is @ <a href="http://review.whamcloud.com/#change,5757" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5757</a></p> <p>Hi, Amir. Nice to meet you.</p> <p>I heartily agree your implementation because I also think reading a file into kernel space can be a kind of dangerous.<br/> But running the new script means that the timing to read a configuration file should be changed. That's why I wonder that lnet configuration will be the same as expected when "lustre_routes_config" or something like that runs on a router node after "lctl network up". Because LNetNIInit() checks "im_a_router" there and, I guess, it may have some effects on interpreting configuration.<br/> Actually, I'm not so sure that it can be problem but I'll appreciate if you answer my question, or consider the case if you've never considered it.</p> <p>Thank you <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> </p> <p>Hi Hiroya,</p> <p>My apologies for the delay, for some reason I wasn't getting any notifications on Jira of comments being added. I will look into that.</p> <p>I have looked at the case that you have indicated; and you are absolutely right. The im_a_router flag will not be set when initially LNetNIInit() is called, and that will cause the "forwarding" parameter not to be checked, resulting in an lnet router node not behaving as expected. Thanks for pointing this out.</p> <p>There are a couple of way to approach this:</p> <p>1. Is to ensure that for a router node, in the luster.conf you enter a routes module param entry that corresponds to the NID of the router. This way when the routes param is being parsed a router node will set the "im_a_router" param properly and thus allocate the pools. The script: "lustre_routes_config" can be used later on to add extra routing entries as desired.<br/> This solution would be temporary until a feature we have in the pipe, DLC, lands. This will allow the modifications of lnet configurations dynamically.</p> <p>2. Is to modify the patch to ensure that when routes are added, if a route NID is added which corresponds to the local NID, then im_a_route is set, and immediately the pools are created.</p> <p>Please let me know your thoughts on these approaches.</p> <p>Thanks again for catching this case.</p> <p>amir</p> <p>Hi Hiroya,</p> <p>A note to highlight:</p> <p>The DLC (Dynamic Lnet Configuration) will be backwards compatible. (IE, existing configurations will not need to be changed, when DLC is introduced)</p> <p>thanks<br/> amir</p> <p>I have updated review: <br/> <a href="http://review.whamcloud.com/#change,5757" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5757</a> <br/> to add the man pages and<br/> <a href="http://review.whamcloud.com/#change,5965" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5965</a><br/> for the testing portion<br/> If you guys can review it, that would be great.</p> <p>Since this change added the ability to configure LNet routes from a file, and updated the lnet init script to load that config file at start time, I think the reload and probe actions should be updated so that they will reload the configuration from the file, if necessary.</p> <p>From the LSB, this is what the reload action should do:</p> <blockquote> <p>cause the configuration of the service to be reloaded without actually stopping and restarting the service</p></blockquote> <p>Similarly probe should do the following (according to SUSE, at least):</p> <blockquote> <p>probe — Probes for the necessity of a reload. Depending on the service, prints "reload" or "restart" if a reload is required. Nothing is printed if a reload is not required.</p></blockquote> <p>I am curious about this because I am working on porting at least the lnet init script to SLES. Amir, am I right to think that the lnet init script should call lustre_routes_config during the reload operation?</p> Related LUDOC-147 Development Rank 1|hzvko7: Rank (Obsolete) 7082