https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2456 Lustre <p>This ticket has been created to track the main development work for the Dynamic LNet Config project.</p> LU-2456

Dynamic LNet Config Main Development Work

New Feature Major Resolved Fixed Amir Shehata Doug Oucharek Tue, 28 Aug 2012 13:39:26 +0000 Mon, 29 May 2017 03:16:56 +0000 Tue, 10 Mar 2015 13:36:21 +0000 Lustre 2.6.0 Lustre 2.7.0 0 20 <p>Project development timeline</p> <p>I've attached a PDF of a timeline I have projected for the development work for this project. It was done with the following criteria:</p> <p>1- One developer is working on the project (me).<br/> 2- Only 80% of my time is included. I'm assuming 20% time on other Jira bugs, etc.<br/> 3- I rounded each line item off to the nearest week.<br/> 4- Each line item will be a separate code patch requiring inspections.<br/> 5- Time has not been allocated for correcting patches based on inspection feedback.<br/> 6- Time has not been allocated for other potential project demands such as FastForward.</p> <p>The first of 5 patches is in Gerrit: <a href="http://review.whamcloud.com/#change,4793" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4793</a></p> <p>The second of 5 patches in Gerrit: <a href="http://review.whamcloud.com/#change,4872" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4872</a></p> <p>3rd patch (route_buffer support) in Gerrit: <a href="http://review.whamcloud.com/#change,4874" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4874</a></p> <p>4th patch (net support) in Gerrit: <a href="http://review.whamcloud.com/#change,4994" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4994</a></p> <p>Previous patches are no longer applicable. I'm in the process of adding new patches.</p> <p>In patch <a href="http://review.whamcloud.com/#/c/8024/" class="external-link" target="_blank" rel="nofollow noopener">8024</a>, Doug said:</p> <blockquote> <p>I'm all for retrieving a src.rpm file during the build, but am not sure the package manager is the right way to go for 3 reasons:</p> <p>1- My understanding is many sites don't have direct access to the internet so how can the interaction with a package manager be integrated seamlessly into a build system? If you have to carry files to a system on a USB key, I fail to see where the package manager buys you anything.</p> <p>2- This goes against the open source freedom ideology, but I don't feel a product as complex, and critical, as Lustre should use a package manager to grab code which is not "exactly" the same as we tested with in-house. I know you can specify version numbers to a package manager, but it seems better overall to just get a src.rpm with all the other build stuff that has been used in all pre-release testing.</p> <p>3- Is there a universal package manager available for all the distro's using Lustre today (does everything support "yum")? What about customers who might be rolling their own vanilla kernel like Fujitsu does?</p></blockquote> <p>Doug, yes, the conversation and examples were mostly about the rpm packaging path, but they should not be intrepretted as requiring rpm. When building rpms, the methods I described should be used. When rpm packaging is not the end goal, folks should be able to build lustre at the command line as they choose. I fully support that.</p> <p>There are is a standard autotools mechanisms for building against external packages. See this section of the autoconf manual:</p> <p><a href="http://www.gnu.org/software/autoconf/manual/autoconf.html#External-Software" class="external-link" target="_blank" rel="nofollow noopener">http://www.gnu.org/software/autoconf/manual/autoconf.html#External-Software</a></p> <p>You guys should add "AC_ARGS_WITH(<span class="error">&#91;libyaml&#93;</span>" handling to Lustre's build system. Then users can either:</p> <ol> <li>Install libyaml in system-standard locations and standard paths (be it through "make install" under the libyaml build tree, or by installing an rpm, or by isntalling a dpkg package, or whatever method they choose). They should not need to specify --with-libyaml=&lt;PATH&gt; in this case.</li> <li>They build libyaml in their home directory or some other nostandard location that does not appear in known compiler and linker paths. They will have to add "--with-libyaml=/path/to/my/fine/libyaml/directory" at lustre's configure time.</li> </ol> <p>This stuff is absolutely simple and standard in the open source software world. Please embrace it. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>November 19th, 2013 </p> <ul class="alternate" type="square"> <li>Attached Feature Test Plan (DLC_FTP.docx)</li> <li>Code Patches are now available<br/> <a href="http://review.whamcloud.com/8020" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8020</a><br/> <a href="http://review.whamcloud.com/8021" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8021</a><br/> <a href="http://review.whamcloud.com/8022" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8022</a><br/> <a href="http://review.whamcloud.com/8023" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8023</a><br/> <a href="http://review.whamcloud.com/8025" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8025</a><br/> <a href="http://review.whamcloud.com/8026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8026</a></li> </ul> <p>Documentation patch:<br/> <a href="http://review.whamcloud.com/8263" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8263</a></p> <p>I think it would be helpful if someone could explain the general plan for all of the new libraries being introduced to the lustre install by this work. For instance, why do we want to see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>libcYAML liblustreconfigapi libauto_complete liblustre_cfg_cmds </pre> </div></div> <p>Without looking into the code too closely, my naive thought is that what we really want is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>liblnet </pre> </div></div> <p>Just one clearly named library. Do we really need four new libraries, at least two of which aren't even very clearly lustre related (from their names)? None of the names identify them as related to lnet configuration.</p> <p>I would also suggest that any new libraries that we introduce need to be properly versioned. That has been sorely lacking in lustre. Using libtool would simplify the make files and provide the options to correctly version the libraries.</p> <p><b>liblustreconfigapi:</b><br/> This library introduces a C-API which abstracts away the interaction with the kernel. Underneath the sheets this API calls ioctl to communicate with the kernel and do the actual work. It introduces two sets of APIs:</p> <ul class="alternate" type="square"> <li>APIs to configure/delete/show routes, networks, buffers</li> <li>APIs that take a YAML file describing the entities that need to be operated on. These YAML blocks are parsed <b>in user space</b> and then the first set of APIs are called appropriately.</li> <li>The design intent is to give tools such as IML (which is python based) an easy way to configure multiple entities, by the use of a YAML text file</li> <li>The YAML file can also be used to configure multiple nodes</li> <li>EX: a node is configured, then you can issue a show configuration command, to dump a YAML representation of the configuration, then take that YAML file and use it to configure newly provisioned nodes or other nodes in the system.</li> </ul> <p><b>libcYAML</b><br/> This library is needed to wrap the libyaml library. The 3rd party libyaml library, provides a YAML stream parsing. Meaning as it parses the YAML file it reports events. What I do is take these events and build and internal representation of the YAML text file. This internal representation is then passed as parameters between functions and used as out parameters for show APIs and error blocks</p> <p><b>libauto_complete</b><br/> Is a new library for the configuration utility introduced. This library provides a way to do CLI auto-completion based on the commands described in a YAML file. The idea is that you're able to reuse this same library for any CLI utility in the future, without having to modify the infrastructure. All you'll need to do is provide a YAML file describing the CLI commands and the action function names that need to be called whenever a command is matched. The library will be able to parse the YAML file describing the commands using the libyaml and libcYAML as described above, and read the commands being typed in, perform auto completion if desired by user (hitting the tab or '?' keys), then once a command is matched and the level of matching in the YAML description file defines a function to be called, the library calls it.</p> <p><b>liblustre_cfg_cmds</b><br/> Is the library which defines the action functions to be called when a command is matched by the libauto_complete library. Multiple of these libraries can be created for different CLIs and the underlying infrastructure need not to be changed.</p> <p>Please note that the design intention is not to restrict DLC to only LNET parameters, but to allow the expansion to configuring other Lustre parameters in the future (if the use case arises). That's why I don't use the term LNET explicitly when I'm naming libraries, since under this design, the same libraries can be expanded to include APIs to configure Lustre parameter.</p> <p>Also all this is explained in greater detail in the user documentation:<br/> <a href="http://review.whamcloud.com/8263" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8263</a></p> <p>As for appropriately versioning the libraries, I was simply following the way the rest of the lustre libraries are being built. I will need to look at how libtool can be used.</p> <p>From that description, it sounds like libauto_complete, libcYAML, and possibly liblustre_cfg_cmds libraries are internal implementation details of the lustre tools, and not appropriate for installation system-wide for users. Perhaps you want what the GNU tools call "convenience" libraries:</p> <p><a href="http://www.gnu.org/software/automake/manual/html_node/Libtool-Convenience-Libraries.html" class="external-link" target="_blank" rel="nofollow noopener">http://www.gnu.org/software/automake/manual/html_node/Libtool-Convenience-Libraries.html</a></p> <blockquote><p>Also all this is explained in greater detail in the user documentation:<br/> <a href="http://review.whamcloud.com/8263" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8263</a></p></blockquote> <p>I will try to provide more review details in the documentation patch, but I may not get to it before some time next week. But some high level comments here:</p> <ul> <li>I think you might have the wrong target audience in mind for this documentation. The graphic that you include should be removed. It mentions things like Chroma - a code name that is not used publically any longer, and is also Intel proprietary software, so not really appropriate for the manual that covers the open source lustre tree. It doesn't really provide any useful information to system administrators, and that is your target audience for this chapter.</li> <li>Details about internal implementation of the library functions are not appropriate for the operation manual.</li> <li>This should not discuss the "dynamic lnet configuration project". The project will soon enough be in the past. The commands that users will use are the focus. As such, this new code should <em>start</em> the lnet configuration section, not appear at the end, and simply be called "LNet Configuration".</li> <li>The <em>first</em> thing you need to talk about is how to use the user-space commands to configure lnet. In fact, I don't think that <em>any</em> talk of APIs is appropriate for "Configuring Lustre Networking (LNET)" chapter of the book. Move the APIs to much later in the book. You could have a one-line note of the form "see chapter X for APIs details" that links to the appropriate API chapter.</li> </ul> <blockquote><p>As for appropriately versioning the libraries, I was simply following the way the rest of the lustre libraries are being built. I will need to look at how libtool can be used.</p></blockquote> <p>That is understandable. Unfortunately, Lustre has a great deal of technical debt that we do not wish to perpetuate in new code.</p> <p>Additionally, I think that the "liblustreconfigapi" name should be changed. This is specifically about configuring <em>lnet</em>, not <em>lustre</em>. Giving the library that name will confuse users and system administrators.</p> <p>Thanks for the documentation feedback, I'll work on another revision of the patch.</p> <p>With regards to the libraries, I don't see them as convenience libraries. These are libraries which are required for both the new configuration tool and the LNET Configuration C-API, and they will need to be installed along with the standard Lustre installation. </p> <p>The libcYAML has to be separate... since it's potentially used by multiple binaries. Integrating it in the C-API or the configuration tool would mean code replication</p> <p>The auto_complete library is infrastructure library. The intent is for it to be used by any future configuration utilities. As it currently stands the new configuration tool is the only one utilizing it, so I can possibly make it as part of the binary (which in earlier iterations of the code it was), however, it is good design to create it as a separate infrastructure entity, so the potential for its reuse is obvious..</p> <p>As I mentioned the intent is for the possibility of extending the liblustreconfigapi to contain also lustre configuration. Making the name change will force us to create a different library, if and when this happens. I'll discuss internally and see which approach we want to take going forward.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>With regards to the libraries, I don't see them as convenience libraries. These are libraries which are required for both the new configuration tool and the LNET Configuration C-API, and they will need to be installed along with the standard Lustre installation.</pre> </div></div> <p>You may be misinterpreting the term "convenience library". It doesn't mean that the code is unimportant. It simply means that the library only exists within the confines of the build tree, and is a "convenience" to the software developers in that they can refer to a group of object files by a single name to use the code.</p> <p>The configuration tool and the "LNET Configuration C-API" are both things that are developed internal to the Lustre tree. Unless we have a very clear use case for applications external to lustre that need to use these internal libraries, they definitely do not belong in /usr/lib.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>As I mentioned the intent is for the possibility of extending the liblustreconfigapi to contain also lustre configuration.</pre> </div></div> <p>I don't remember that being a stated goal of this project. Did I miss that? Does the design document explain how that would be done? I thought that the goal of this project was to make configuring lnet more like you configure other networks on Linux systems. To make the networking configuration cleaner, more like sysadmins are used to, and dynamic.</p> <p>I'm rather uneasy about that goal of configuring lnet and lustre from one unified interface. We have long kept a clean separation between lnet and the rest of lustre, and we need to be very careful about throwing that away. I for one would prefer that we keep lnet and lustre configuration separate.</p> <p>At LLNL, we even have two separate and distinct init files, one for starting lnet, and another for starting lustre servers.</p> <p>The new configuration tool, like lctl will be in /usr/sbin. The tool needs to link against these libraries. Also let's say a 3rd party tool is developed to configure LNET, such as IML, it will need to link against the LNET Configuration Library which will pull in the libcYAML, as an example. From my reading on Convenience Library: <a href="http://www.sourceware.org/autobook/autobook/autobook_74.html#SEC74" class="external-link" target="_blank" rel="nofollow noopener">http://www.sourceware.org/autobook/autobook/autobook_74.html#SEC74</a><br/> These are entities which are neither a library nor an executable, but will subsequently need to be linked into either in order for it to be useful. this means to me that when convenience library get linked, they become part of the entity they are linked against. Instead of specifying a bunch of .o files in your gcc command you just specify that name of the convenience library and then that gets expanded into the .o files. This is not the intention of these libraries. They are intended to be standalone shared libraries, which will linked dynamically against an executable. </p> <p>The other concern I understand is you're saying that since they do not have a use outside of lustre, then they shouldn't be in /usr/lib. Where would they be? For example both liblustre and liblustreapi are in /usr/lib64, and both of these libraries are intended for use in lustre only. Can you please clarify this point.</p> <p>These look like interesting libraries and they might potential for reuse, but I think most users would be quite surprised to see the lustre package install seemingly extraneous utility libraries into /usr/lib. If these libraries are indeed more generally useful, then the proper thing to do is move them to their own project and publish them as a standalone package. For now, though, I suggest just linking these libraries into the one tool that is using them (as already suggested), and we can consider whether or not we publish them as standalone libraries independently of this discussion. </p> <p>Next time, just write the CLI in Python. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>after discussing it with Robert, I more fully understand the reasons for not installing these libraries in /usr/lib. I'll change it to eliminate the installation of these libraries. However, as liblustreconfigapi is intended to be used externally, same as liblustre and liblustreapi, it will still be installed as it is now. However, I see your point about separation between lustre and lnet and will make the appropriate changes.</p> <p>The consideration I want to add is the need for the new API libraries to be LGPL rather than GPL. Third party applications which are not GPL'ed cannot use these libraries if they have to be GPL. If the way these libraries are built into our existing binaries dictate they be GPL, then that won't work.</p> <p>Just a nit, but would you mind dropping the "api" from then name of that library? It's a bit like saying "PIN number"</p> <p>+1 for LGPL. There is already a precedent for this as the new HSM library functions were helpfully added by CEA as LGPL. (Though they are distributed in liblustreapi, with is of course mostly GPL. d'oh!)</p> <blockquote><p>after discussing it with Robert, I more fully understand the reasons for not installing these libraries in /usr/lib. I'll change it to eliminate the installation of these libraries. However, as liblustreconfigapi is intended to be used externally, same as liblustre and liblustreapi, it will still be installed as it is now.</p></blockquote> <p>Awesome! I'm glad we are on the same page. I do understand that liblustreconfigapi should be public, which is why I only listed the other three as a problem. My only issue with liblustreconfigapi was the name. Robert makes a good point about dropping "api" from the name as well. We did that for "liblustreapi", but it did feel dirty when I did it. The rationale was that "liblustre" name was already taken in the lustre world. So you can add liblustreapi to your list of lustre counter-examples. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>After getting feedback from Andreas, Robert and Chris, I believe the best way to move forward, is to make the LNET configuration utility use the existing lctl infrastructure. This will naturally eliminate all libraries except for the LNET C Configuration API (the LGPL library). The utility will be a thin layer, which parses commands line input and directly call into the LNET C Configuration API, without doing any error checking, since all error checking is localized in the library.</p> <p>Additionally, I'll move the LNET C Configuration API Library and the LNET configuration utility into the lnet/utils directory, to maintain a strict separation between LNET and Lustre. </p> <p>Let me know if this plans sounds good to everyone.</p> <p>That's disappointing since lctl is hardly a shining example of CLI design, and it looked like you had some good ideas that would could have used to upgrade the rest of of our tools. However, it's your call. There's always python...</p> <p>BTW, I think having error checking in the library is fine just as long as the library is only returning error codes to the caller and <b>not</b> printing any error messages itself. </p> <blockquote><p>After getting feedback from Andreas, Robert and Chris, I believe the best way to move forward, is to make the LNET configuration utility use the existing lctl infrastructure.</p></blockquote> <p>Actually, I'm not sure that I am in favor of that at all. Right now lnet is pretty much a stand-alone component. lustre has dependencies on lnet, but lnet does not depend on lustre. If you make lnet's configuration reliant on lctl, you introduce a new dependency from lnet on lustre.</p> <p>I think we would like to maintain the clear separation of code and tools between lnet and lustre.</p> <p>The core command line Parser code is actually in libcfs, and is already being used by tools in lnet/utils so this is at least not adding any new dependencies. </p> <p>OK, granted.</p> <p>Still, I am not not in favor of adding this to lctl. I would like to see it as a separate command under the lnet subdirectory somewhere (lnet/utils, for instance).</p> <p>Agreed, and I believe Amir does intend to create a new config utility in lnet. The difference is he now plans to use the same cli parser library that lctl uses, instead of using the new one he originally wrote for DLC. </p> <p>That is correct. I'm almost done my updates to the DLC patches.</p> <p>I would like to argue again for seeing an updated DLC design document attached to this ticket (or in confluence or something).</p> <p>One thing I'm realizing is that I'm not particularly in favor of the command line design. The design currently looks something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetcfg {add|del|show} OBJECT [options]</pre> </div></div> <p>I think that is fundamentally backwards. Instead, we should take inspiration from the linux "ip" command, and other commands like git. The basic model should be:</p> <p>So for instance the first line of the ip command's man page synopsis shows this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ip [ OPTIONS ] OBJECT { COMMAND | help }</pre> </div></div> <p>That is the style that I think we want to emulate for the new command. So some examples might be:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl network add o2ib0 dev ib0 lnetctl route add o2ib9 routers 192.168.121.[160-163]@o2ib2 </pre> </div></div> <p>If there are global variables that we want to set, it is reasonable to use global get/set commands. For instance:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl set small_buffers 1000 lnetctl set medium_buffers 1000 lnetctl set large_buffers 1000 lnetctl set routing 1 lnetctl set routing 0 </pre> </div></div> <p>For handling yaml files, I think the original plan is kind of ugly:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl add file --name &lt;yaml file name&gt; lnetctl del file --name &lt;yaml file name&gt; </pre> </div></div> <p>I can't even really wrap my head around how the "del file" behavior would even work. For instance if I have this section in my yaml file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>buffer: enable: 0 </pre> </div></div> <p>and then I feed it "del file", do I wind up with routing enabled? If I have this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>buffer: tiny: 2000 </pre> </div></div> <p>What does "del file" do??? If I can't figure this out, there is no way that a sysadmin will.</p> <p>And "buffer" is the wrong name for that yaml section as well. But I digress.</p> <p>A better command line for this might look like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl import &lt; myfinefile.yaml lnetctl import myfinefile.yaml lnetctl export &gt; backupsettings.yaml lnetctl export backupsettings.yaml </pre> </div></div> <p>The CLI</p> <p>lnetctl &lt;cmd&gt; &lt;entity&gt; <span class="error">&#91;parameters&#93;</span></p> <p>except that we use optional parameters to pass in the different parameters. The decision is to get away from positional parameters.</p> <p>ex:</p> <p>add route --net tcp0 --gateway 10.1.1.2@tcp1 --hop 1 --priority 0</p> <p>Also please note that the examples you gave above are not consistent form. In some cases you have </p> <p>lnetctl &lt;command&gt; &lt;object&gt;<br/> ex: lnetctl set small_buffers 1000</p> <p>and in other cases you have</p> <p>lnetctl OBJECT COMMAND<br/> ex: lnetctl network add o2ib0 dev ib0</p> <p>In my opinion, I think a consistent form makes it easier to get more familiar with the command line.</p> <p>I agree that "del file" with regards to enabling/disabling routing might be confusing. Here is my suggestion.</p> <p>To reset all router buffers to default values the yaml file that you feed to the del command would look like:</p> <ul class="alternate" type="square"> <li>buffer</li> </ul> <p>If you add other parameters under buffer they will be ignored. For example:</p> <ul class="alternate" type="square"> <li>buffer:<br/> tiny: 2000</li> </ul> <p>it'll still reset the buffer to the default values.</p> <p>To disable routing the yaml file would include:</p> <ul class="alternate" type="square"> <li>routing</li> </ul> <p>This way you can take the exact same YAML file feed it to any of the commands and they will get processed correctly. Parameters that are meaningless to the command are ignored.</p> <p>With your suggestion:<br/> lnetctl import myfinefile.yaml</p> <p>What operations do we do on myfinefile.yaml? What is contained in the yaml file?</p> <p>That's why you explicitly tell the utility what to do on the yaml file (add, del or show)</p> <p>User Documentation as exported from the internal Wiki</p> <p>I have a recommendation. I'm finding that progressing on the project with this form of communication is inefficient. Would you agree for a skype/phone meeting, where we can hash out the different points of view. I would say that we can call a meeting with yourself, Doug Oucharek, Andreas and myself. Does that sound like a reasonable plan?</p> <p>I'm up for a call. I am free after 3:30 PST today, or most of the day tomorrow.</p> <p>comments for DLC patch <a href="http://review.whamcloud.com/#/c/9785/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9785/</a></p> <p>suggestions about dlc APIs based on discussion in Miami</p> <p><a href="http://review.whamcloud.com/#/c/9830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9830</a><br/> <a href="http://review.whamcloud.com/#/c/9831" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9831</a><br/> <a href="http://review.whamcloud.com/#/c/9832" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9832</a></p> <p><a href="http://review.whamcloud.com/#/c/8021" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8021</a><br/> <a href="http://review.whamcloud.com/#/c/8022" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8022</a><br/> <a href="http://review.whamcloud.com/#/c/8023" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8023</a><br/> <a href="http://review.whamcloud.com/#/c/8025" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8025</a><br/> <a href="http://review.whamcloud.com/#/c/8026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8026</a><br/> <a href="http://review.whamcloud.com/#/c/8027" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8027</a><br/> <a href="http://review.whamcloud.com/#/c/8028" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8028</a></p> <p>I have been testing this the utilities and I saw this:</p> <p>lnetctl &gt; help set<br/> set: set </p> {tiny_buffers | small_buffers | large_buffers | routing} <p>lnetctl &gt; help set large_buffers<br/> setlarge_buffers: Unknown command.</p> <p>Is this correct behavior?</p> <p>lnetctl utility has the same work flow as the ip utility. The only difference is that the infrastructure I was asked to use for the utility allows interactive mode, and the patter "help &lt;cmd&gt;" in interactive mode. However, that infrastructure doesn't support the pattern "help &lt;cmd&gt; &lt;subcmd&gt;" readily.</p> <p>The way it works:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lnetctl &gt; help set set: set {tiny_buffers | small_buffers | large_buffers | routing} lnetctl &gt; set tiny_buffers help set tiny_buffers: set tiny routing buffers VALUE must be greater than 0 </pre> </div></div> <p>so help &lt;cmd&gt; gives the first level of help. To get help on subsequent commands, you put the "help" keyword after the subcommand.</p> <p>If you run directly from the command line, the output would look like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@rt1 lutfTMP]# lnetctl help Available commands are: lnet route net routing set <span class="code-keyword">import</span> export stats peer_credits help exit quit For more help type: help command-name [root@rt1 lutfTMP]# lnetctl route help route add: add a route --net: net name (ex tcp0) --gateway: gateway nid (ex 10.1.1.2@tcp) --hop: number to <span class="code-keyword">final</span> destination (1 &lt; hops &lt; 255) --priority: priority of route (0 - highest prio route del: delete a route --net: net name (ex tcp0) --gateway: gateway nid (ex 10.1.1.2@tcp) route show: show routes --net: net name (ex tcp0) to filter on --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on --hop: number to <span class="code-keyword">final</span> destination (1 &lt; hops &lt; 255) to filter on --priority: priority of route (0 - highest prio to filter on --detail: display detailed output per route route help: display <span class="code-keyword">this</span> help [root@rt1 lutfTMP]# lnetctl route show help route show: show routes --net: net name (ex tcp0) to filter on --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on --hop: number to <span class="code-keyword">final</span> destination (1 &lt; hops &lt; 255) to filter on --priority: priority of route (0 - highest prio to filter on --detail: display detailed output per route </pre> </div></div> <p>One question that came up at LAD last week was why the <tt>lnetctl</tt> functionality was not (also) available via <tt>lctl</tt>? Like the other LNet-specific commands that are already available via lctl, these commands could also be added just by adding them to the lctl parser.</p> <p>It is still desirable to have a standalone lnetctl command for the systems that are using only LNet (there are a few). </p> <p>The stated benefit of having these new commands in lctl is that this reduces the number of binaries that need to be installed on the compute nodes, which try to have a minimal memory/space footprint.</p> <p>That isn't a requirement for landing, but rather an enhancement request from the end users.</p> <p>Ticket <a href="https://jira.whamcloud.com/browse/LU-5705" title="Dynamic LNet Config: merge lnetctl into lctl" class="issue-link" data-issue-key="LU-5705">LU-5705</a> was created to discuss lctl versus lnetctl</p> <p>Ticket <a href="https://jira.whamcloud.com/browse/LU-5712" title="Derive LGPL library from libcfs" class="issue-link" data-issue-key="LU-5712"><del>LU-5712</del></a> was created to look at creating a new LGPL library from the parts of libcfs (GPL) used by this feature.</p> <p>Test plan attached is dated, please update the document. </p> <p>Landed in 2.7</p> Duplicate LU-1071 Related LU-1279 LU-5734 LU-5568 LU-5838 LU-5849 LU-5854 LU-5861 LU-5874 LU-5875 LU-5884 LUDOC-204 LU-5098 LU-5850 LU-2457 LU-2633 Development Rank 1|hzvdj3: Rank (Obsolete) 5798