[LU-10214] Example `lnet_routes.conf` syntax for gateway range is not supported

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.1
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.5
• Type: Bug
• Priority: Major
• Reporter: Stanford Research Computing Center
• Assignee: Sonia Sharma (Inactive)
• Resolution: Fixed

Environment

CentOS 7

• Severity: 3

Description

The default lnet_routes.conf file provided by the lustre-client-2.10.1-1.el7.x86_64 RPM contains an example using an IP range syntax to specify multiple gateways, that doesn't seem to be supported by lnetctl:

tcp7: { gateway: 10.3.3.[6-12]@tcp, priority: 20, hop: 8 }

When feeding that example file to lustre_routes_config, the following error is generated:

# lustre_routes_config  /etc/lnet_routes.conf
Can't parse gateway NID "10.3.3.[6-12]@tcp"

The easy fix would be to remove the example from the configuration file, but a much better one would be to make {lnetctl}} accept IP ranges.

The previous way of handling lnet configuration via kernel module options supported the syntax, so it appears like lnetctl not supporting it is a regression.

Comments

[Amir Shehata (Inactive)]

lustre_routes_config doesn't use lnetctl, it uses lctl. You're right that lnetctl doesn't support IP ranges for routes. We can add that capability, if need be. But for your specific issue it seems to be something else. will investigate and update the ticket.

[Stanford Research Computing Center]

Hi Amir, 

 

You're right, lustre_routes_config uses lnet not lnet_ctl.

 

With the single following line in /etc/lnet_routes.conf:

o2ib1:  { gateway: 10.9.0.[31-32]@o2ib4 }

the output of lustre_routes_config is:

Can't parse gateway NID "10.9.0.[31-32]@o2ib4"

Tracing the bash script shows that it executes the following command, that generates this error:

+ lctl --net o2ib1 add_route '10.9.0.[31-32]@o2ib4' 
Can't parse gateway NID "10.9.0.[31-32]@o2ib4" 
+ RC=1

So, lctl doesn't seem to be supporting IP ranges.

 

But it doesn't seem like lnetctl supports it either:

# lnetctl route add --net o2ib1 --gateway 10.9.0.[31-32]@o2ib4 
add: 
    - route: 
          errno: -1 
          descr: "cannot parse gateway NID '10.9.0.[31-32]@o2ib4'"

 

Anyway, my point is two-fold:

1. Neither lnetctl nor lctl support IP  ranges when adding routes, which is a regression compared to the previous way of defining routes via lnet kernel module options,

2. The default lnet_routes.conf file provided in the lustre-client RPM contains a non-working syntax example, which is very confusing.

 

Would it be possible to add back IP range support to define routes via lctl or lnetctl?

 

Thanks!

[Gerrit Updater]

Sonia Sharma ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30076
Subject: LU-10214 lnet: Add support for gateway range
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2076f24978bf1cf491c44765548d0c2b554d703f

[Amir Shehata (Inactive)]

Please note that "lustre_routes_config" script would've never worked with address ranges, since lctl never supported that from userspace. Only the module parameters supported that. So there is no regression in that sense. I agree that there is a discrepancy between how userspace configuration and kernel module parameters work.

Going forward though, I would really like to add new features to the lnetctl utility only, since that's intended to be the primary utility for configuring LNet. And lustre_routes_config script will eventually be deprecated. Can you use lnetctl for configuring routes?

[Stanford Research Computing Center]

Yes, absolutely, using lnetctl would be completely fine.

Actually, we would like to be able to use the lnet service and the provided systemd unit files. I believe that at this time, the route configuration part is commented out in the unit file, so I guess it would have to be adapted. But any way to let us specify IP range in a configuration file that the lnet service would use to conigure the routes would be fine by us.

Thanks!

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30511
Subject: LU-10214 lnet: allow expressions for route config
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4d4577e6fe47614a9e24b55053c6e5d4a09bf560

[Amir Shehata (Inactive)]

I added the feature that you requested. Please try it out and let me know if it fits your needs. Note I only added the feature to lnetctl.

so you can configure routes as follows:

lnetct route add --net <net> --gateway <expr-list>.<expr-list>.<expr-list>.<expr-list>
# example
lnetctl route add --net tcp1 --gateway 192.18.[93-94].[1-12,23-25]@tcp

or via YAML

route:
    - net: <net>
      gateway: <expr-list>.<expr-list>.<expr-list>.<expr-list>

The syntax supported is identical to the ip2net syntax.

[Stanford Research Computing Center]

Thank you! We'll test  and get back to you, but this looks awesome.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30511/
Subject: LU-10214 lnet: allow expressions for route config
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 235e8dea46f6f9ec0021c789cb5b9214cee94e38

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32248
Subject: LU-10214 lnet: allow expressions for route config
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 39380603e53bc7c60adc1d8887af3f6f0511fee1

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32248/
Subject: LU-10214 lnet: allow expressions for route config
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 8c610a5f483d53de7d9579587f304b551c847e99