[LU-2456] Dynamic LNet Config Main Development Work

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.6.0
• Assignee: Amir Shehata (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

This ticket has been created to track the main development work for the Dynamic LNet Config project.

Issue Links

• is duplicated by: LU-1071 able to handle many lnet routes
• is related to: LU-5734 LNet dynamic control: lnet_dyn_add_ni() can't clean up failed NI in some cases
• is related to: LU-5568 kernel crash when when network initialization failed
• is related to: LU-5838 DLC: inconsistent nameing of seqno vs seq_no
• is related to: LU-5849 DLC: parse networks function needs to return the number of parsed networks
• is related to: LU-5854 DLC: lnetctl import failed as Failed to handle token:2 [state=0, rc=-1]
• is related to: LU-5861 DLC: lnet startup script doesn't invoke lnetctl properly
• is related to: LU-5874 DLC: the ongoing traffic was interrupted after adding a new network interface
• is related to: LU-5875 DLC: failed adding an existing network interface when there is traffic ongoing
• is related to: LU-5884 bad lnet conf causes LBUG
• is related to: LUDOC-204 Update Lustre Manual for Dynamic LNET Config
• is related to: LU-5098 /proc/sys/lnet/routes is stupid
• is related to: LU-5850 DLC: lnet_startup_lndnis should clean up after itself
• is related to: LU-1279 failure trying to mount two targets at the same time after boot

Comments

[Andreas Dilger]

One question that came up at LAD last week was why the lnetctl functionality was not (also) available via lctl? Like the other LNet-specific commands that are already available via lctl, these commands could also be added just by adding them to the lctl parser.

It is still desirable to have a standalone lnetctl command for the systems that are using only LNet (there are a few).

The stated benefit of having these new commands in lctl is that this reduces the number of binaries that need to be installed on the compute nodes, which try to have a minimal memory/space footprint.

That isn't a requirement for landing, but rather an enhancement request from the end users.

[Amir Shehata (Inactive)]

lnetctl utility has the same work flow as the ip utility. The only difference is that the infrastructure I was asked to use for the utility allows interactive mode, and the patter "help <cmd>" in interactive mode. However, that infrastructure doesn't support the pattern "help <cmd> <subcmd>" readily.

The way it works:

lnetctl > help set
set: set {tiny_buffers | small_buffers | large_buffers | routing}

lnetctl > set tiny_buffers help
set tiny_buffers: set tiny routing buffers
        VALUE must be greater than 0

so help <cmd> gives the first level of help. To get help on subsequent commands, you put the "help" keyword after the subcommand.

If you run directly from the command line, the output would look like this:

[root@rt1 lutfTMP]# lnetctl help
Available commands are:
        lnet
        route
        net
        routing
        set
        import
        export
        stats
        peer_credits
        help
        exit
        quit
For more help type: help command-name
[root@rt1 lutfTMP]# lnetctl route help
route add: add a route
        --net: net name (ex tcp0)
        --gateway: gateway nid (ex 10.1.1.2@tcp)
        --hop: number to final destination (1 < hops < 255)
        --priority: priority of route (0 - highest prio

route del: delete a route
        --net: net name (ex tcp0)
        --gateway: gateway nid (ex 10.1.1.2@tcp)

route show: show routes
        --net: net name (ex tcp0) to filter on
        --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
        --hop: number to final destination (1 < hops < 255) to filter on
        --priority: priority of route (0 - highest prio to filter on
        --detail: display detailed output per route

route help: display this help

[root@rt1 lutfTMP]# lnetctl route show help
route show: show routes
        --net: net name (ex tcp0) to filter on
        --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
        --hop: number to final destination (1 < hops < 255) to filter on
        --priority: priority of route (0 - highest prio to filter on
        --detail: display detailed output per route

[James A Simmons]

I have been testing this the utilities and I saw this:

lnetctl > help set
set: set

{tiny_buffers | small_buffers | large_buffers | routing}

lnetctl > help set large_buffers
setlarge_buffers: Unknown command.

Is this correct behavior?

[James A Simmons]

http://review.whamcloud.com/#/c/9830
http://review.whamcloud.com/#/c/9831
http://review.whamcloud.com/#/c/9832

http://review.whamcloud.com/#/c/8021
http://review.whamcloud.com/#/c/8022
http://review.whamcloud.com/#/c/8023
http://review.whamcloud.com/#/c/8025
http://review.whamcloud.com/#/c/8026
http://review.whamcloud.com/#/c/8027
http://review.whamcloud.com/#/c/8028

[Liang Zhen (Inactive)]

suggestions about dlc APIs based on discussion in Miami

[Liang Zhen (Inactive)]

comments for DLC patch http://review.whamcloud.com/#/c/9785/

[Christopher Morrone (Inactive)]

I'm up for a call. I am free after 3:30 PST today, or most of the day tomorrow.

[Amir Shehata (Inactive)]

I have a recommendation. I'm finding that progressing on the project with this form of communication is inefficient. Would you agree for a skype/phone meeting, where we can hash out the different points of view. I would say that we can call a meeting with yourself, Doug Oucharek, Andreas and myself. Does that sound like a reasonable plan?

[Amir Shehata (Inactive)]

User Documentation as exported from the internal Wiki

[Amir Shehata (Inactive)]

The CLI

lnetctl <cmd> <entity> [parameters]

except that we use optional parameters to pass in the different parameters. The decision is to get away from positional parameters.

ex:

add route --net tcp0 --gateway 10.1.1.2@tcp1 --hop 1 --priority 0

Also please note that the examples you gave above are not consistent form. In some cases you have

lnetctl <command> <object>
ex: lnetctl set small_buffers 1000

and in other cases you have

lnetctl OBJECT COMMAND
ex: lnetctl network add o2ib0 dev ib0

In my opinion, I think a consistent form makes it easier to get more familiar with the command line.

I agree that "del file" with regards to enabling/disabling routing might be confusing. Here is my suggestion.

To reset all router buffers to default values the yaml file that you feed to the del command would look like:

• buffer

If you add other parameters under buffer they will be ignored. For example:

• buffer:
  tiny: 2000

it'll still reset the buffer to the default values.

To disable routing the yaml file would include:

• routing

This way you can take the exact same YAML file feed it to any of the commands and they will get processed correctly. Parameters that are meaningless to the command are ignored.

With your suggestion:
lnetctl import myfinefile.yaml

What operations do we do on myfinefile.yaml? What is contained in the yaml file?

That's why you explicitly tell the utility what to do on the yaml file (add, del or show)

[Christopher Morrone (Inactive)]

I would like to argue again for seeing an updated DLC design document attached to this ticket (or in confluence or something).

One thing I'm realizing is that I'm not particularly in favor of the command line design. The design currently looks something like this:

lnetcfg {add|del|show} OBJECT [options]

I think that is fundamentally backwards. Instead, we should take inspiration from the linux "ip" command, and other commands like git. The basic model should be:

So for instance the first line of the ip command's man page synopsis shows this:

ip [ OPTIONS ] OBJECT { COMMAND | help }

That is the style that I think we want to emulate for the new command. So some examples might be:

lnetctl network add o2ib0 dev ib0
lnetctl route add o2ib9 routers 192.168.121.[160-163]@o2ib2

If there are global variables that we want to set, it is reasonable to use global get/set commands. For instance:

lnetctl set small_buffers 1000
lnetctl set medium_buffers 1000
lnetctl set large_buffers 1000
lnetctl set routing 1
lnetctl set routing 0

For handling yaml files, I think the original plan is kind of ugly:

lnetctl add file --name <yaml file name>
lnetctl del file --name <yaml file name>

I can't even really wrap my head around how the "del file" behavior would even work. For instance if I have this section in my yaml file:

buffer:
        enable: 0

and then I feed it "del file", do I wind up with routing enabled? If I have this:

buffer:
        tiny: 2000

What does "del file" do??? If I can't figure this out, there is no way that a sysadmin will.

And "buffer" is the wrong name for that yaml section as well. But I digress.

A better command line for this might look like:

lnetctl import < myfinefile.yaml
lnetctl import myfinefile.yaml
lnetctl export > backupsettings.yaml
lnetctl export backupsettings.yaml