LNet currently does not support IPv6. A presentation was done by Isaac Huang at this year's LUG talking about the large challenge of supporting IPv6. No one has taken ownership of this task yet (I believe it was on the OpenSFS list of projects to fund).
Doug it seems reasonable that we don't support IPv6, but we also probably shouldn't kernel panic just because we see an IPv6 address.
Indeed the desired outcome from this issue is to fix the panic 
In fact when the interface had only an IPv6 address, Lustre handled it gracefully:
LNetError: 3588:0:(linux-tcpip.c:137:libcfs_ipif_query()) Can't get IP address for interface ib0 LNetError: 3588:0:(o2iblnd.c:2569:kiblnd_create_dev()) Can't query IPoIB interface ib0: -99 LNetError: 105-4: Error -100 starting up LNI o2ib LustreError: 3588:0:(events.c:737:ptlrpc_init_portals()) network initialisation failed
I believe our IO nodes on the LLNL BGQ clusters have both IPv4 and IPv6 addresses and we didn't run into this, so I'm not sure why it was a problem on IBM's system.
I wonder if this is an endian issue (since the IBM system is PPC64)? Are any of the IO nodes on the LLNL BGQ cluster PPC?
Yes they are all PPC.
Personally, I wouldn't call those messages on the console as "handled gracefully"; but it's better than a panic. If the issue is IPv6 compatibility, stating that on the console would be nice:
LNetError: ib0 configured for IPv6, which is not supported by LNet. Network initialization failed as a result.
Good point. It was only the EADDRNOTAVAIL return value that suggested it might be an IPv6 issue, but that's not exactly user-friendly.
I suspect we got NULL pointer (cmid->device) at here:
kiblnd_dev_failover()->ib_alloc_pd(cmid->device);
I'm wondering how it could happen because we did check returned value for all calls, and we also specified both address and AF_INET before calling rdma_bind_addr(), which can attach cmid to device:
rdma_bind_addr()->cma_acquire_dev()->cma_acquire_dev()->cma_attach_to_dev()
Perhaps address scope is a factor. IBM's system had a site-scoped v6 address whereas the LLNL systems have only automatically derived link-local addresses.
When we do a kinlnd_create_dev(), it calls libcfs_ipif_query() to get the address which is used in the rdma_bind_addr(). If libcfs_ipif_query() were to get an IPv6 address (struct sockaddr_in6) rather than an IPv4 address (struct sockaddr_in), then we would misinterpret what was returned and pass a badly formed address to rdma_bind_addr(). I'm wondering if that is what happened here.
In kiblnd_create_dev(), we make use of the ioctl SIOGIFADDR with the address type set to AF_INET. In theory, this should only return an IPv4 address; we need to use AF_INET6 for IPv6. There seems to be a lot of confusion in networking forums as to how SIOGIFADDR works which is made worse by the fact that the IPv6 struct, sockaddr_in6, does not fit into the generic sockaddr structure which is returned by SIOGIFADDR.
I will need to look at the specific source code implementation of SIOGIFADDR for the distro IBM is running.
Ned: what distro and version is IBM running?
Liang: Another way we can do this is to use SIOGIFCONF to get a list of all interfaces and addresses. That way we can be smart in our error messages when we can see there is only an IPv6 address. However, due to the structure size differences I mention above, some implementations do not return IPv6 addresses at all.
Ned: what distro and version is IBM running?
seqio33-ib0@root:uname -a Linux seqio33-ib0 2.6.32-220.23.3.bgq.el6_V1R1M2_0.ppc64 #1 SMP Sat Aug 25 18:22:10 CDT 2012 ppc64 ppc64 ppc64 GNU/Linux seqio33-ib0@root:cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.2 (Santiago)
Fix on master: http://review.whamcloud.com/#change,3815
Doug, was that a fix? I thought it was just a debugging patch. Perhaps I've missed something.
Whoops! My bad. Doing bulk cleanup of Jira tickets and did not see "debug patch" in patch title.
Change 3815 is only a debug patch and not a fix.