https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1211 Lustre <p>We are having problems installing lustre with Myrinet support on a customer site.<br/> The building process seems fine, the MX drivers work standalone (we can load the drivers, bring up interfaces, set IP address and get communication with other servers). We also manage to build lustre with no warning or error messages.<br/> However, when installing the RPMs a bunch of kmxlnd.ko messages pops up concerning about unknown symbols, like mx_*</p> <p>This is the process we are following:</p> <p>1) Files we are using:<br/> kernel-headers-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm<br/> kernel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm<br/> lustre-source-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf.x86_64.rpm<br/> kernel-debuginfo-common-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm<br/> kernel-devel-2.6.18-274.3.1.el5_lustre.g9500ebf.x86_64.rpm<br/> mx_1.2.12.tar.gz</p> <p>2) Install the kernel and lustre source, and reboot<br/> rpm <del>Uvh --nodeps kernel</del>* lustre-source-*<br/> reboot</p> <p>3) build the MX driver<br/> ./configure --enable-kernel-lib --enable-10g --enable-ether-mode<br/> make rpm<br/> rpm -Uvh mx-1.2.12-1.x86_64.rpm</p> <p>4) Build Lustre<br/> ./configure --enable-quota --with-server --disable-lru-resize --enable-ext4 --disable-health-write --with-mx=/root/mx/mx-1.2.12</p> <p>make rpms</p> <p>cd /usr/src/redhat/RPMS/x86_64/<br/> rpm -Uvh lustre-1.8.7* lustre-ldiskfs* lustre-modules*</p> <p>All lustre packages get installed but these warning messages pops up:<br/> ... &lt;snip&gt;<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_open_endpoint<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_finalize<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_iconnect<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strerror<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_endpoint_addr_context<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kirecv<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr_context<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wait_any<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_nic_id_to_board_number<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_close_endpoint<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx__init_api<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_register_unexp_handler<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kisend<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_test_any<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_decompose_endpoint_addr<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strstatus<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_cancel<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_request_timeout<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wakeup<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_connect<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_decompose_endpoint_addr2<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_disconnect<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_open_endpoint<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_finalize<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_iconnect<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_strerror<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_set_endpoint_addr_context<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_kirecv<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_get_endpoint_addr_context<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_wait_any<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_nic_id_to_board_number<br/> WARNING: /lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko needs unknown symbol mx_close_endpoint<br/> ... &lt;/snip&gt;</p> <p>5) Bring up the MX driver:<br/> /opt/mx/sbin/mx_start_stop start<br/> Loading mx driver<br/> Creating mx devices</p> <p>6) Bringing up kmxlnd<br/> modprobe kmxlnd<br/> FATAL: Error inserting kmxlnd (/lib/modules/2.6.18-274.3.1.el5_lustre.g9500ebf/updates/kernel/net/lustre/kmxlnd.ko): Unknown symbol in module, or unknown parameter (see dmesg)</p> <p>And the same kind of messages are also logged on dmesg:<br/> ...&lt;snip&gt;<br/> kmxlnd: Unknown symbol mx_get_endpoint_addr<br/> kmxlnd: Unknown symbol mx_open_endpoint<br/> kmxlnd: Unknown symbol mx_finalize<br/> kmxlnd: Unknown symbol mx_iconnect<br/> kmxlnd: Unknown symbol mx_strerror<br/> kmxlnd: Unknown symbol mx_set_endpoint_addr_context<br/> kmxlnd: Unknown symbol mx_kirecv<br/> ...&lt;/snip&gt;</p> <p>So, Does anyone at WC could help us to figure out what's wrong here and how can we make this configuration work?<br/> Some sanity checklist or install guide maybe!?</p> <p>Thank you.</p> Lustre 1.8.7-WC1 <br/> OS server version: Red Hat Enterprise Linux Server release 5.5 (Tikanga) - DDN EXASCALER 1.5.0 OS Image<br/> 2 x MDS Intel E5645 @ 2.4Ghz (6 core / Westmere), 48GB RAM<br/> 8 x OSS Intel E5645 @ 2.4Ghz (6 core / Westmere), 48GB RAM<br/> Myrinet Myri10G dual protocol NIC (rev 01)<br/> LU-1211

Issues building and installing Lustre 1.8.7-wc1 with MYRINET support

Bug Major Resolved Fixed Cliff White Carlos Thomaz paj server Tue, 13 Mar 2012 12:53:20 +0000 Sun, 29 Apr 2012 12:24:07 +0000 Sun, 29 Apr 2012 12:24:07 +0000 Lustre 1.8.7 0 6 <p>Just a remark about MX driver building process:<br/> 1) we followed lustre manuals and created a link between common and include <br/> cd &lt;path&gt;/mx<br/> ln -s common include</p> <p>2) we tried different MX builds:<br/> 2.1) ./configure --with-kernel-lib <br/> 2.2) ./configure --with-kernel-lib --enable-10g --enable-ether-mode</p> <p>3) when building lustre with MX support we also tried to point out to /opt/mx but that fails.<br/> ./configure --enable-quota --with-server --disable-lru-resize --enable-ext4 --disable-health-write --with-mx=/opt/mx <br/> FAILED!</p> <p>Cliff will help with this one</p> <p>Okay, it's been a bit since we've had customers using Myrinet, I am researching which versions we support. <br/> Can you re-do the build and capture all the build output and attach to this bug? </p> <p>Cliff, Please find attached the compressed script file with the full output from the entire process described in here.<br/> Thank you<br/> Carlos</p> <p>Could you please check if the symbols which are reported as missing actually exist in /proc/kallsyms once the mx module is loaded?</p> <p>Hi Johan. We are rebuilding the server at this moment. We found something... The DDN RHEL ISO bundled up the lustre kernel 2.6.18-238.12.1.el5_lustre.gce5e033. The following RPMs are installed by default:<br/> kernel-2.6.18-238.12.1.el5_lustre.gce5e033<br/> ddn-lustre-tools-0.5-2012.01.18.130500<br/> kernel-headers-2.6.18-238.12.1.el5_lustre.gce5e033<br/> lustre-modules-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.gce5e033<br/> lustre-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.gce5e033<br/> kernel-devel-2.6.18-238.12.1.el5_lustre.gce5e033<br/> lustre-ldiskfs-3.1.50-wc1_2.6.18_238.12.1.el5_lustre.gce5e033<br/> ddn-ofed-1.5.3.1-2.6.18_238.12.1.el5_lustre.gce5e033</p> <p>When installing the new RPMs from whamcloud (in order to rebuild lustre with MX support) If, instead of deleting the old rpms (rpm -e), we just upgrade them (rpm -Uhv), we manage to build lustre successfully. </p> <p>So, these are the differences on the packages: OSS5 is a default installation, OSS7 is the installation with MX modules.</p> <p><span class="error">&#91;root@oss07 ~&#93;</span># rpm -qa |grep lustre | sort<br/> ddn-lustre-tools-0.5-2012.01.18.130500<br/> ddn-ofed-1.5.3.1-2.6.18_238.12.1.el5_lustre.gce5e033<br/> kernel-2.6.18-274.3.1.el5_lustre.g9500ebf<br/> kernel-devel-2.6.18-274.3.1.el5_lustre.g9500ebf<br/> kernel-headers-2.6.18-274.3.1.el5_lustre.g9500ebf<br/> lustre-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf<br/> lustre-ldiskfs-3.1.51-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf<br/> lustre-modules-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf<br/> lustre-source-1.8.7-wc1_2.6.18_274.3.1.el5_lustre.g9500ebf<br/> <span class="error">&#91;root@oss07 ~&#93;</span># ssh oss05 rpm -qa |grep lustre| sort<br/> ddn-lustre-tools-0.5-2012.01.18.130500<br/> ddn-ofed-1.5.3.1-2.6.18_238.12.1.el5_lustre.gce5e033<br/> kernel-2.6.18-238.12.1.el5_lustre.gce5e033<br/> kernel-devel-2.6.18-238.12.1.el5_lustre.gce5e033<br/> kernel-headers-2.6.18-238.12.1.el5_lustre.gce5e033<br/> lustre-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.gce5e033<br/> lustre-ldiskfs-3.1.50-wc1_2.6.18_238.12.1.el5_lustre.gce5e033<br/> lustre-modules-1.8.6-wc1_2.6.18_238.12.1.el5_lustre.gce5e033</p> <p>Our guess is, if you try to install a server from the scratch using the lustre packages g9500ebc it will not work. </p> <p>The MX driver we are using is 1.2.12</p> <p>We'll continue working in this and we'll update this ticket as soon as we get some news.</p> <p>Thanks.</p> <p>hi, did you depmod after installing the modules?</p> <p>Liang</p> <p>Hi Liang. Yes, we did and looks allright.</p> <p>We managed to get things working and finished the building process. As explained before, seems like there some conflicts with files when upgrading some RPMs or when adding the lustre source RPM. We don't quite figure out yet what exactly is causing the problem since we are running late in this deployment. However, our plan is to continue to investigate it and understand why it happens. </p> <p>Another question for now is about the MX compatibility mode. The servers (OSS and MDS) has MX cards and may run natively, but the clients are Gbit ethernet. As far as I understand we should run Lustre on tcp mode since that's the only thing the clients can communicate. However, when building the MX driver with ether-support (--enable-ether-mode --enable-10g-mode) the network seems to stop responding. We know this is not a lustre issue, but we are wondering if anyone has some suggestions on how to build the driver.</p> <p>It's also interesting that even building the driver with no ethernet mode or 10g mode support we still able to bringup the interface, assign an IP address. This is an output from mx_info when built with ethernet mode and 10g mode support</p> <p><span class="error">&#91;root@oss05 to-install&#93;</span># /opt/mx/bin/mx_info <br/> MX Version: 1.2.12<br/> MX Build: root@oss08:/root/mx_ether/mx-1.2.12 Wed Mar 14 11:39:25 CDT 2012<br/> 2 Myrinet boards installed.<br/> The MX driver is configured to support a maximum of:<br/> 8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host<br/> ===================================================================<br/> Instance #0: 364.4 MHz LANai, PCI-E x8, 2 MB SRAM<br/> Status: Running, P0: Wrong Network<br/> Network: Ethernet 10G</p> <p> MAC Address: 00:60:dd:45:1a:20<br/> Product code: 10G-PCIE2-8B2L-2QP<br/> Part number: 09-04247<br/> Serial number: 427870<br/> Mapper: 00:60:dd:45:1a:21, version = 0x00000000, configured<br/> Mapped hosts: 1</p> <p> ROUTE COUNT<br/> INDEX MAC ADDRESS HOST NAME P0<br/> ----- ----------- --------- &#8212;<br/> 0) 00:60:dd:45:1a:20 oss05:0 1,0<br/> 1) 00:60:dd:45:1a:21 oss05:1 D 0,0<br/> ===================================================================<br/> Instance #1: 364.4 MHz LANai, PCI-E x8, 2 MB SRAM<br/> Status: Running, P0: Wrong Network<br/> Network: Ethernet 10G</p> <p> MAC Address: 00:60:dd:45:1a:21<br/> Product code: 10G-PCIE2-8B2L-2QP<br/> Part number: 09-04247<br/> Serial number: 427870<br/> Mapper: 00:60:dd:45:1a:21, version = 0x00000000, configured<br/> Mapped hosts: 1</p> <p> ROUTE COUNT<br/> INDEX MAC ADDRESS HOST NAME P0<br/> ----- ----------- --------- &#8212;<br/> 0) 00:60:dd:45:1a:20 oss05:0 D 0,0<br/> 1) 00:60:dd:45:1a:21 oss05:1 1,0<br/> <span class="error">&#91;root@oss05 to-install&#93;</span># </p> <p>We can't see the other host indexes, mac and host names and also there's a clear message on status line saying "Wrong message" .</p> <p>this is how my modprobe.conf line looks like:<br/> options lnet networks=mx0(myri0)</p> <p>PS: We still capable to lctl ping. </p> <p>Thanks<br/> Carlos</p> <p>Hi Carlos, if it said "Wrong network", does it mean that the network adapter is connected to wrong port on a switch? I think we see this kind of error only if MX-10G was configured with "Ethernet mode" but the network adapter is connected to a 10G Myrinet switch port instead of a 10G Ethernet switch port on a switch.</p> <p>This is a piece of information I just found:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MX-10G supports both MXoM (MX over Myrinet) and MXoE (MX over Ethernet). Refer to What is MX/Ethernet? for further explanation. MXoM assumes that the Myri-10G Network Adapters are being used in "Myrinet mode" and are connected to 10G Myrinet switch ports on a Myri-10G switch. MXoE assumes that the Myri-10G Network Adapters are being used in "Ethernet mode" and are connected to either a 10GbE switch or to the 10GbE switch ports on a Myri-10G switch." The output of the MX tool mx_info reports the status of the firmware on the NIC as well as the status of the link connectivity. If you see a message similar to: Status: Running, P0: Wrong Network Network: Ethernet 10G then the Network: output indicates that MX-10G was configured with --enable-ether-mode for MXoE support. However, since Status reports Wrong Network, this indicates that the network adapters are erroneously connected to a 10G Myrinet switch port instead of a 10G Ethernet switch port. To correct this error, you would then need to either reconfigure MX or connect the cables to a different switch. </pre> </div></div> <p>Hi there</p> <p>Is there any further action needed on this ticket?</p> <p>Please advise</p> <p>Peter</p> <p>I am going to close this issue, please reopen if you have more information</p> Development Epic server Rank 1|hzvf5b: Rank (Obsolete) 6099 Severity