[LU-2456] Dynamic LNet Config Main Development Work

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.6.0
• Assignee: Amir Shehata (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

This ticket has been created to track the main development work for the Dynamic LNet Config project.

Issue Links

• is duplicated by: LU-1071 able to handle many lnet routes
• is related to: LU-5734 LNet dynamic control: lnet_dyn_add_ni() can't clean up failed NI in some cases
• is related to: LU-5568 kernel crash when when network initialization failed
• is related to: LU-5838 DLC: inconsistent nameing of seqno vs seq_no
• is related to: LU-5849 DLC: parse networks function needs to return the number of parsed networks
• is related to: LU-5854 DLC: lnetctl import failed as Failed to handle token:2 [state=0, rc=-1]
• is related to: LU-5861 DLC: lnet startup script doesn't invoke lnetctl properly
• is related to: LU-5874 DLC: the ongoing traffic was interrupted after adding a new network interface
• is related to: LU-5875 DLC: failed adding an existing network interface when there is traffic ongoing
• is related to: LU-5884 bad lnet conf causes LBUG
• is related to: LUDOC-204 Update Lustre Manual for Dynamic LNET Config
• is related to: LU-5098 /proc/sys/lnet/routes is stupid
• is related to: LU-5850 DLC: lnet_startup_lndnis should clean up after itself
• is related to: LU-1279 failure trying to mount two targets at the same time after boot

Comments

[James A Simmons]

Ticket LU-5705 was created to discuss lctl versus lnetctl

[Andreas Dilger]

One question that came up at LAD last week was why the lnetctl functionality was not (also) available via lctl? Like the other LNet-specific commands that are already available via lctl, these commands could also be added just by adding them to the lctl parser.

It is still desirable to have a standalone lnetctl command for the systems that are using only LNet (there are a few).

The stated benefit of having these new commands in lctl is that this reduces the number of binaries that need to be installed on the compute nodes, which try to have a minimal memory/space footprint.

That isn't a requirement for landing, but rather an enhancement request from the end users.

[Amir Shehata (Inactive)]

lnetctl utility has the same work flow as the ip utility. The only difference is that the infrastructure I was asked to use for the utility allows interactive mode, and the patter "help <cmd>" in interactive mode. However, that infrastructure doesn't support the pattern "help <cmd> <subcmd>" readily.

The way it works:

lnetctl > help set
set: set {tiny_buffers | small_buffers | large_buffers | routing}

lnetctl > set tiny_buffers help
set tiny_buffers: set tiny routing buffers
        VALUE must be greater than 0

so help <cmd> gives the first level of help. To get help on subsequent commands, you put the "help" keyword after the subcommand.

If you run directly from the command line, the output would look like this:

[root@rt1 lutfTMP]# lnetctl help
Available commands are:
        lnet
        route
        net
        routing
        set
        import
        export
        stats
        peer_credits
        help
        exit
        quit
For more help type: help command-name
[root@rt1 lutfTMP]# lnetctl route help
route add: add a route
        --net: net name (ex tcp0)
        --gateway: gateway nid (ex 10.1.1.2@tcp)
        --hop: number to final destination (1 < hops < 255)
        --priority: priority of route (0 - highest prio

route del: delete a route
        --net: net name (ex tcp0)
        --gateway: gateway nid (ex 10.1.1.2@tcp)

route show: show routes
        --net: net name (ex tcp0) to filter on
        --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
        --hop: number to final destination (1 < hops < 255) to filter on
        --priority: priority of route (0 - highest prio to filter on
        --detail: display detailed output per route

route help: display this help

[root@rt1 lutfTMP]# lnetctl route show help
route show: show routes
        --net: net name (ex tcp0) to filter on
        --gateway: gateway nid (ex 10.1.1.2@tcp) to filter on
        --hop: number to final destination (1 < hops < 255) to filter on
        --priority: priority of route (0 - highest prio to filter on
        --detail: display detailed output per route

[James A Simmons]

I have been testing this the utilities and I saw this:

lnetctl > help set
set: set

{tiny_buffers | small_buffers | large_buffers | routing}

lnetctl > help set large_buffers
setlarge_buffers: Unknown command.

Is this correct behavior?

[James A Simmons]

http://review.whamcloud.com/#/c/9830
http://review.whamcloud.com/#/c/9831
http://review.whamcloud.com/#/c/9832

http://review.whamcloud.com/#/c/8021
http://review.whamcloud.com/#/c/8022
http://review.whamcloud.com/#/c/8023
http://review.whamcloud.com/#/c/8025
http://review.whamcloud.com/#/c/8026
http://review.whamcloud.com/#/c/8027
http://review.whamcloud.com/#/c/8028

[Liang Zhen (Inactive)]

suggestions about dlc APIs based on discussion in Miami

[Liang Zhen (Inactive)]

comments for DLC patch http://review.whamcloud.com/#/c/9785/

[Christopher Morrone (Inactive)]

I'm up for a call. I am free after 3:30 PST today, or most of the day tomorrow.

[Amir Shehata (Inactive)]

I have a recommendation. I'm finding that progressing on the project with this form of communication is inefficient. Would you agree for a skype/phone meeting, where we can hash out the different points of view. I would say that we can call a meeting with yourself, Doug Oucharek, Andreas and myself. Does that sound like a reasonable plan?

[Amir Shehata (Inactive)]

User Documentation as exported from the internal Wiki

[Amir Shehata (Inactive)]

The CLI

lnetctl <cmd> <entity> [parameters]

except that we use optional parameters to pass in the different parameters. The decision is to get away from positional parameters.

ex:

add route --net tcp0 --gateway 10.1.1.2@tcp1 --hop 1 --priority 0

Also please note that the examples you gave above are not consistent form. In some cases you have

lnetctl <command> <object>
ex: lnetctl set small_buffers 1000

and in other cases you have

lnetctl OBJECT COMMAND
ex: lnetctl network add o2ib0 dev ib0

In my opinion, I think a consistent form makes it easier to get more familiar with the command line.

I agree that "del file" with regards to enabling/disabling routing might be confusing. Here is my suggestion.

To reset all router buffers to default values the yaml file that you feed to the del command would look like:

• buffer

If you add other parameters under buffer they will be ignored. For example:

• buffer:
  tiny: 2000

it'll still reset the buffer to the default values.

To disable routing the yaml file would include:

• routing

This way you can take the exact same YAML file feed it to any of the commands and they will get processed correctly. Parameters that are meaningless to the command are ignored.

With your suggestion:
lnetctl import myfinefile.yaml

What operations do we do on myfinefile.yaml? What is contained in the yaml file?

That's why you explicitly tell the utility what to do on the yaml file (add, del or show)