[LU-17889] Autotest nodes should not use 'fe80:' link local (loopback) IPv6 address

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Affects Version/s: Lustre 2.16.0
• Environment: Maloo test bed nodes
• Severity: 3
• Assignee: Michael Kvardakov
• Reporter: James A Simmons

Description

Pushing patches to run sanity-lnet test with IPv6 which are working locally for me to maloo revealed that the IPv6 address setup is only local link (fe80:.....). These addresses are only visible to the local node so the more complex LNet test can't run. Also LNet ignores these addresses. Proper IPv6 addresses need to be setup.

Issue Links

• is related to: LU-17367 Failover on master: Invalid NID string
• is related to: LU-16822 test IPv6 NIDs in sanity-lnet.sh

Comments

[Charlie Olmstead]

Michael pushed a patch to correct the IPV6 issue (https://review.whamcloud.com/#/c/private/lab/+/55584/)

I retested one of the sessions from patch https://review.whamcloud.com/#/c/fs/lustre-release/+/55435/3 and the fd33 address is now listed:

/usr/sbin/lnetctl net show
net:
-     net type: lo
      local NI(s):
      -     nid: 0@lo
            status: up
-     net type: tcp
      local NI(s):
      -     nid: 10.240.24.248@tcp
            status: up
            interfaces:
                  0: eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:72:a5:1e brd ff:ff:ff:ff:ff:ff
    altname enp0s3f0
    altname ens3f0
    inet 10.240.24.248/20 brd 10.240.31.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fd33:3981:3213:f010:0:5254:72:a51e/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe72:a51e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: test1pl@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:a2:28:ad:93:2b brd ff:ff:ff:ff:ff:ff link-netns test_ns
    inet6 fe80::3ca2:28ff:fead:932b/64 scope link tentative 
       valid_lft forever preferred_lft forever 

 

[Andreas Dilger]

Michael, I'm not an IPv6 expert, I was only passing on information given to me from another call on this topic.  I think the main interest for this ticket from James & Chris is that the test nodes have proper IPv6 addresses configured, and they can work out issues in the test script itself.

I would suggest to use the most recent version of the IPv6 test patch.

As for why Janitor is running tests on your patch, this is normal for all patches submitted to Gerrit.

[Charlie Olmstead]

mkvardakov - chef does run on all test nodes. AT deletes all chef data after ljb runs.

[Michael Kvardakov]

The more I diving to the issue the more questions I get.
First, adilger I updated the patch in line with your comments, and lsanity-test does not seem to run anymore. Probably that caused by whole block handling LARGE_ADDR_ENABLE lustre/tests/test-framework.sh is missing in master somehow. So it run without -l option now:

/usr/sbin/lnetctl lnet configure -l

Second, after comparing successful and failed runs, I noticed that

/usr/sbin/lnetctl net add --nid @tcp
---
add:
-     net:
      errno: -22
      descr: ! "unsupported NID"
...

is wrong. It should take valid ip address like:

/lnet/utils/lnetctl net add --nid 192.168.202.8@tcp

Third, I don't quite understand why tests started by my commits run on some oleg* servers in 192.168.* network

Fourth, colmstea last but not least, how come chef is not configured at target nodes throughout all runs? They just don't have /etc/chef/client.rb and I cannot check why ipv6 is not configured:
a) ipv6 is disabled completely in initial test runs
b) as expected in recent test runs

[Michael Kvardakov]

thanks adilger, now I can see that test node did not get ipv6 address as expected. Have retriggered test session, investigating

[Andreas Dilger]

mkvardakov, looking at the following test runs from James' patch:
https://testing.whamcloud.com/test_sets/3c1e5454-23e2-4162-958c-af443ed4fde3
https://testing.whamcloud.com/test_sets/9f145a47-09f3-4494-a094-d27ea661de38

I see the following listed near the start of the sanity-lnet.suite_log.onyx-116vm1.log file, from the output of "do_lnetctl net show" and "ip a":

/usr/sbin/lnetctl lnet configure -all --large
/usr/sbin/lnetctl net show
net:
-     net type: lo
      local NI(s):
      -     nid: 0@lo
            status: up
-     net type: tcp
      local NI(s):
      -     nid: 10.240.29.128@tcp
            status: up
            interfaces:
                  0: eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:c3:da:fa brd ff:ff:ff:ff:ff:ff
    altname enp0s3f0
    altname ens3f0
    inet 10.240.29.128/20 brd 10.240.31.255 scope global dynamic eth0
       valid_lft 21113sec preferred_lft 21113sec
    inet6 fe80::5054:ff:fec3:dafa/64 scope link 
       valid_lft forever preferred_lft forever
3: test1pl@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:7f:44:90:4e:ec brd ff:ff:ff:ff:ff:ff link-netns test_ns
    inet6 fe80::30aa:ddff:feb8:56b9/64 scope link tentative 
       valid_lft forever preferred_lft forever

so it looks like the inet6 address assignments are still using the link-local fe80:: range instead of the fd33:3981:3213: ranges that you mention above for onyx and trevis? Is there something that needs to be specified for an IPv6 test session to assign these non-local addresses? Is there a reason that shouldn't be done automatically for all test sessions?

[Minh Diep]

ssmirnov, could you please advise to what we do next?

[James A Simmons]

test1pg is used to create a fake IP address for testing. It doesn't match to anything real. So the 2001:db8:0:f101::1@tcp is correct. The above is a very special test case.

[Michael Kvardakov]

simmonsja cannot reproduce it, getting different lnectl ping error. If I understand correctly, --source should take as local ipv6 address which starts with fd33:3981:3213:f010 at onyx and fd33:3981:3213:f020 at trevis. Could we review how this parameter is generated?

[Michael Kvardakov]

/usr/sbin/lnetctl net add --net tcp --if test1pg
default via 10.240.16.1 dev eth0
NI 0@lo expect status "up" found "up"
NI 10.240.25.232@tcp expect status "up" found "up"
NI 2001:db8:0:f101::1@tcp expect status "up" found "up"
/usr/sbin/lnetctl ping --source 2001:db8:0:f101::1@tcp 10.240.25.232@tcp
manage:
- ping:
  errno: -5
  descr: ! 'failed to ping 10.240.25.232@tcp: Input/output error'
 sanity-lnet test_214: @@@@@@ FAIL: /usr/sbin/lnetctl ping --source 2001:db8:0:f101::1@tcp 10.240.25.232@tcp failed 

[James A Simmons]

we shall see what fails