[LU-11893] doesn't handle logical network interface properly.

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Environment: 2.12
• Severity: 3
• Assignee: James A Simmons
• Reporter: Shuichi Ihara

Description

 # ifconfig | grep ib
 Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
 Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
 Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
 Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
 infiniband 20:00:10:86:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
 ib0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
 infiniband 20:00:10:86:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
 ib1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
 infiniband 20:00:18:86:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)
 ib1:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044
 infiniband 20:00:18:86:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)

Lustre-2.10.5 works well

 # cat /etc/modprobe.d/lustre.conf 
 options lnet networks="o2ib0(ib0), o2ib1(ib0:0), o2ib2(ib1), o2ib3(ib1:0)"
 # modprobe lustre

 Jan 28 12:52:17 ai200-7f94-vm00 kernel: LNet: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 4
 Jan 28 12:52:17 ai200-7f94-vm00 kernel: alg: No test for adler32 (adler32-zlib)
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: Lustre: Lustre: Build Version: 2.10.5_ddn7_2_g7fd8383
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: LNet: Using FastReg for registration
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: LNet: Added LNI 172.16.251.20@o2ib [8/256/0/180]
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: LNet: Added LNI 172.16.252.20@o2ib1 [8/256/0/180]
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: LNet: Added LNI 172.16.253.20@o2ib2 [8/256/0/180]
 Jan 28 12:52:18 ai200-7f94-vm00 kernel: LNet: Added LNI 172.16.254.20@o2ib3 [8/256/0/180]

lustre-2.12 doesn't handle logical interface properly

# modprobe lustre
modprobe: ERROR: could not insert 'lustre': Network is down

Jan 28 13:00:56 ai200-7f94-vm00 kernel: LNet: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 4
Jan 28 13:00:56 ai200-7f94-vm00 kernel: alg: No test for adler32 (adler32-zlib)
Jan 28 13:00:57 ai200-7f94-vm00 kernel: Lustre: Lustre: Build Version: 2.12.0
Jan 28 13:00:57 ai200-7f94-vm00 kernel: LNet: Using FastReg for registration
Jan 28 13:00:57 ai200-7f94-vm00 kernel: LNet: Added LNI 172.16.251.20@o2ib [8/256/0/180]
Jan 28 13:00:57 ai200-7f94-vm00 kernel: LNetError: 6305:0:(lib-socket.c:105:lnet_ipif_query()) Can't get flags for interface ib0:0
Jan 28 13:00:57 ai200-7f94-vm00 kernel: LNetError: 6305:0:(o2iblnd.c:2879:kiblnd_create_dev()) Can't query IPoIB interface ib0:0: -19
Jan 28 13:00:57 ai200-7f94-vm00 kernel: LNetError: 105-4: Error -100 starting up LNI o2ib
Jan 28 13:00:58 ai200-7f94-vm00 kernel: LNet: Removed LNI 172.16.251.20@o2ib
Jan 28 13:00:58 ai200-7f94-vm00 kernel: LustreError: 6305:0:(events.c:625:ptlrpc_init_portals()) network initialisation failed

Issue Links

• is related to: LU-12269 Support RHEL 8.0
• is related to: LU-12381 o2iblnd uses wrong IB interface
• is related to: LU-12511 Prepare lustre for adoption into the linux kernel
• is related to: LU-11838 Support linux kernel version 4.18
• is related to: LU-6399 Libcfs socket cleanup

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34993/
Subject: LU-11893 lnet: consoldate secondary IP address handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b770d7117f35a972bd2c9ffef03a17dbcb036d20

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35442
Subject: LU-11893 lnet: consoldate secondary IP address handling
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 3d3ae9946b67c66ee9ce8a7e916d899e8a2b7197

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35248/
Subject: LU-11893 o2iblnd: add secondary IP address handling
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 4d863145dd63d387e138bb20dcf2d5f1b66a52aa

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35159/
Subject: LU-11893 ksocklnd: add secondary IP address handling
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: eb46e3374193895b01fe16a8975553c87133a52e

[James A Simmons]

I got positive feedback from Chris Horn. Looks like https://review.whamcloud.com/#/c/34993 is ready for reviews.

[James A Simmons]

I think I resolved the ip2net string issues.

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35248
Subject: LU-11893 o2iblnd: add secondary IP address handling
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 3034ad8fdb7fda499ded48313ab9f0479d063188

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35159
Subject: LU-11893 ksocklnd: add secondary IP address handling
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 9f98099692cf4bfc18000226ac09bee3be2d6e74

[Peter Jones]

The countdown continues- one to go

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34476/
Subject: LU-11893 o2iblnd: add secondary IP address handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c4b39bf56bbcacd49d7f888a0745cd4b5580b36b

[James A Simmons]

Two patches left.