[LU-17889] Autotest nodes should not use 'fe80:' link local (loopback) IPv6 address

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Affects Version/s: Lustre 2.16.0
• Environment: Maloo test bed nodes
• Severity: 3
• Assignee: Michael Kvardakov
• Reporter: James A Simmons

Description

Pushing patches to run sanity-lnet test with IPv6 which are working locally for me to maloo revealed that the IPv6 address setup is only local link (fe80:.....). These addresses are only visible to the local node so the more complex LNet test can't run. Also LNet ignores these addresses. Proper IPv6 addresses need to be setup.

Issue Links

• is related to: LU-17367 Failover on master: Invalid NID string
• is related to: LU-16822 test IPv6 NIDs in sanity-lnet.sh

Comments

[Andreas Dilger]

I've asked that a more clear error message is added to mount.lustre when a link-local fe80: address is used, but something should likely be added into lnetctl or similar to print a very clear message like "Unsupported link-local IPv6 NID type 'fe80:...'" instead of just returning "-22" in that case, so this can be caught very early during configuration instead of later on when these addresses are being used.

[Michael Kvardakov]

simmonsja

[root@onyx-79vm20 ~]# ip a          
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ce:6b:b5 brd ff:ff:ff:ff:ff:ff
    altname enp0s3f0
    altname ens3f0
    inet 10.240.26.175/20 brd 10.240.31.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fd33:3981:3213:f010:0:5254:ce:6bb5/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fece:6bb5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@onyx-79vm20 ~]# lctl net up -l
Writer error: failed to resolve Netlink family id
LNET configure error 22: (null)
[root@onyx-79vm20 ~]# lnetctl lnet configure -l
Writer error: failed to resolve Netlink family id
---
configure:
-     lnet:
      errno: 0
      descr: ! "LNet configure error: (null)"
...

[James A Simmons]

LU-16822 hasn't landed yet so no IPv6 by default. You can do lnetctl lnet configure -l or lctl net up -l to see if fd33:XXX ends up in lnetctl net  show

[Charlie Olmstead]

Michael pushed a patch to correct the IPV6 issue (https://review.whamcloud.com/#/c/private/lab/+/55584/)

I retested one of the sessions from patch https://review.whamcloud.com/#/c/fs/lustre-release/+/55435/3 and the fd33 address is now listed:

/usr/sbin/lnetctl net show
net:
-     net type: lo
      local NI(s):
      -     nid: 0@lo
            status: up
-     net type: tcp
      local NI(s):
      -     nid: 10.240.24.248@tcp
            status: up
            interfaces:
                  0: eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:72:a5:1e brd ff:ff:ff:ff:ff:ff
    altname enp0s3f0
    altname ens3f0
    inet 10.240.24.248/20 brd 10.240.31.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fd33:3981:3213:f010:0:5254:72:a51e/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe72:a51e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: test1pl@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:a2:28:ad:93:2b brd ff:ff:ff:ff:ff:ff link-netns test_ns
    inet6 fe80::3ca2:28ff:fead:932b/64 scope link tentative 
       valid_lft forever preferred_lft forever 

 

[Andreas Dilger]

Michael, I'm not an IPv6 expert, I was only passing on information given to me from another call on this topic.  I think the main interest for this ticket from James & Chris is that the test nodes have proper IPv6 addresses configured, and they can work out issues in the test script itself.

I would suggest to use the most recent version of the IPv6 test patch.

As for why Janitor is running tests on your patch, this is normal for all patches submitted to Gerrit.

[Charlie Olmstead]

mkvardakov - chef does run on all test nodes. AT deletes all chef data after ljb runs.

[Michael Kvardakov]

The more I diving to the issue the more questions I get.
First, adilger I updated the patch in line with your comments, and lsanity-test does not seem to run anymore. Probably that caused by whole block handling LARGE_ADDR_ENABLE lustre/tests/test-framework.sh is missing in master somehow. So it run without -l option now:

/usr/sbin/lnetctl lnet configure -l

Second, after comparing successful and failed runs, I noticed that

/usr/sbin/lnetctl net add --nid @tcp
---
add:
-     net:
      errno: -22
      descr: ! "unsupported NID"
...

is wrong. It should take valid ip address like:

/lnet/utils/lnetctl net add --nid 192.168.202.8@tcp

Third, I don't quite understand why tests started by my commits run on some oleg* servers in 192.168.* network

Fourth, colmstea last but not least, how come chef is not configured at target nodes throughout all runs? They just don't have /etc/chef/client.rb and I cannot check why ipv6 is not configured:
a) ipv6 is disabled completely in initial test runs
b) as expected in recent test runs

[Michael Kvardakov]

thanks adilger, now I can see that test node did not get ipv6 address as expected. Have retriggered test session, investigating

[Andreas Dilger]

mkvardakov, looking at the following test runs from James' patch:
https://testing.whamcloud.com/test_sets/3c1e5454-23e2-4162-958c-af443ed4fde3
https://testing.whamcloud.com/test_sets/9f145a47-09f3-4494-a094-d27ea661de38

I see the following listed near the start of the sanity-lnet.suite_log.onyx-116vm1.log file, from the output of "do_lnetctl net show" and "ip a":

/usr/sbin/lnetctl lnet configure -all --large
/usr/sbin/lnetctl net show
net:
-     net type: lo
      local NI(s):
      -     nid: 0@lo
            status: up
-     net type: tcp
      local NI(s):
      -     nid: 10.240.29.128@tcp
            status: up
            interfaces:
                  0: eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:c3:da:fa brd ff:ff:ff:ff:ff:ff
    altname enp0s3f0
    altname ens3f0
    inet 10.240.29.128/20 brd 10.240.31.255 scope global dynamic eth0
       valid_lft 21113sec preferred_lft 21113sec
    inet6 fe80::5054:ff:fec3:dafa/64 scope link 
       valid_lft forever preferred_lft forever
3: test1pl@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:7f:44:90:4e:ec brd ff:ff:ff:ff:ff:ff link-netns test_ns
    inet6 fe80::30aa:ddff:feb8:56b9/64 scope link tentative 
       valid_lft forever preferred_lft forever

so it looks like the inet6 address assignments are still using the link-local fe80:: range instead of the fd33:3981:3213: ranges that you mention above for onyx and trevis? Is there something that needs to be specified for an IPv6 test session to assign these non-local addresses? Is there a reason that shouldn't be done automatically for all test sessions?

[Minh Diep]

ssmirnov, could you please advise to what we do next?

[James A Simmons]

test1pg is used to create a fake IP address for testing. It doesn't match to anything real. So the 2001:db8:0:f101::1@tcp is correct. The above is a very special test case.