[LU-1211] Issues building and installing Lustre 1.8.7-wc1 with MYRINET support

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 1.8.7
• Labels: paj server

Environment

Lustre 1.8.7-WC1
OS server version: Red Hat Enterprise Linux Server release 5.5 (Tikanga) - DDN EXASCALER 1.5.0 OS Image
2 x MDS Intel E5645 @ 2.4Ghz (6 core / Westmere), 48GB RAM
8 x OSS Intel E5645 @ 2.4Ghz (6 core / Westmere), 48GB RAM
Myrinet Myri10G dual protocol NIC (rev 01)

• Severity: 1

Epic

server

• Assignee: Cliff White (Inactive)
• Reporter: Carlos Thomaz

Description

We are having problems installing lustre with Myrinet support on a customer site.
The building process seems fine, the MX drivers work standalone (we can load the drivers, bring up interfaces, set IP address and get communication with other servers). We also manage to build lustre with no warning or error messages.
However, when installing the RPMs a bunch of kmxlnd.ko messages pops up concerning about unknown symbols, like mx_*

This is the process we are following:

1) Files we are using:
kernel-headers-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
kernel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
lustre-source-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm
kernel-debuginfo-common-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
kernel-devel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm
mx_1.2.12.tar.gz

2) Install the kernel and lustre source, and reboot
rpm Uvh --nodeps kernel* lustre-source-*
reboot

3) build the MX driver
./configure --enable-kernel-lib --enable-10g --enable-ether-mode
make rpm
rpm -Uvh mx-1.2.12-1.x86_64.rpm

4) Build Lustre
./configure --enable-quota --with-server --disable-lru-resize --enable-ext4 --disable-health-write --with-mx=/root/mx/mx-1.2.12

make rpms

cd /usr/src/redhat/RPMS/x86_64/
rpm -Uvh lustre-1.8.7* lustre-ldiskfs* lustre-modules*

All lustre packages get installed but these warning messages pops up:
... <snip>
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_open_endpoint
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_finalize
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_iconnect
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strerror
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_endpoint_addr_context
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kirecv
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr_context
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wait_any
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_nic_id_to_board_number
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_close_endpoint
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx__init_api
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_register_unexp_handler
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kisend
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_test_any
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_decompose_endpoint_addr
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strstatus
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_cancel
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_request_timeout
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wakeup
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_connect
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_decompose_endpoint_addr2
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_disconnect
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_open_endpoint
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_finalize
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_iconnect
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strerror
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_endpoint_addr_context
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kirecv
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr_context
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wait_any
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_nic_id_to_board_number
WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_close_endpoint
... </snip>

5) Bring up the MX driver:
/opt/mx/sbin/mx_start_stop start
Loading mx driver
Creating mx devices

6) Bringing up kmxlnd
modprobe kmxlnd
FATAL: Error inserting kmxlnd (/lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko): Unknown symbol in module, or unknown parameter (see dmesg)

And the same kind of messages are also logged on dmesg:
...<snip>
kmxlnd: Unknown symbol mx_get_endpoint_addr
kmxlnd: Unknown symbol mx_open_endpoint
kmxlnd: Unknown symbol mx_finalize
kmxlnd: Unknown symbol mx_iconnect
kmxlnd: Unknown symbol mx_strerror
kmxlnd: Unknown symbol mx_set_endpoint_addr_context
kmxlnd: Unknown symbol mx_kirecv
...</snip>

So, Does anyone at WC could help us to figure out what's wrong here and how can we make this configuration work?
Some sanity checklist or install guide maybe!?

Thank you.

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA