https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6083 Lustre <p>I compiled Lustre client under Ubuntu 14.04, with network tcp/ip, it works, but<br/> IB didn't.</p> <p>Download IB driver for ubuntu14(kernel 3.13.0-32-generic) from:<br/> <a href="http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&amp;mrequest=downloads&amp;mtype=ofed&amp;mver=MLNX_OFED-2.3-2.0.0&amp;mname=MLNX_OFED_LINUX-2.3-2.0.0-ubuntu14.04-x86_64.iso" class="external-link" target="_blank" rel="nofollow noopener">http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&amp;mrequest=downloads&amp;mtype=ofed&amp;mver=MLNX_OFED-2.3-2.0.0&amp;mname=MLNX_OFED_LINUX-2.3-2.0.0-ubuntu14.04-x86_64.iso</a></p> <p>Attachment is failed config.log, failed messages firstly come:<br/> "<br/> /usr/src/mlnx-ofed-kernel-2.3/include/linux/compat-2.6.h:17:35: fatal error: linux/compat_autoconf.h: No such file or directory<br/> #include &lt;linux/compat_autoconf.h&gt;<br/> "<br/> And if i skipped this error by remove this including in source file, i still hit following error:</p> <p>"<br/> configure: error: an external source tree was specified for o2iblnd however I could not find a /usr/src/mlnx-ofed-kernel-2.3/Module.symvers there<br/> "<br/> if i touched a Module.symvers(a little hack) there and compile finished, and i installed these debs, when modprobe lustre with IB, i hit following messages:</p> <p><span class="error">&#91;422278.843073&#93;</span> ko2iblnd: Unknown symbol rdma_create_qp (err -22)<br/> <span class="error">&#91;422278.843080&#93;</span> ko2iblnd: disagrees about version of symbol ib_destroy_cq<br/> <span class="error">&#91;422278.843081&#93;</span> ko2iblnd: Unknown symbol ib_destroy_cq (err -22)<br/> <span class="error">&#91;422278.843084&#93;</span> ko2iblnd: disagrees about version of symbol rdma_create_id<br/> <span class="error">&#91;422278.843085&#93;</span> ko2iblnd: Unknown symbol rdma_create_id (err -22)<br/> <span class="error">&#91;422278.843101&#93;</span> ko2iblnd: disagrees about version of symbol rdma_listen<br/> <span class="error">&#91;422278.843103&#93;</span> ko2iblnd: Unknown symbol rdma_listen (err -22)<br/> <span class="error">&#91;422278.843105&#93;</span> ko2iblnd: disagrees about version of symbol rdma_destroy_qp<br/> <span class="error">&#91;422278.843107&#93;</span> ko2iblnd: Unknown symbol rdma_destroy_qp (err -22)<br/> <span class="error">&#91;422278.843113&#93;</span> ko2iblnd: disagrees about version of symbol ib_query_device<br/> <span class="error">&#91;422278.843115&#93;</span> ko2iblnd: Unknown symbol ib_query_device (err -22)<br/> <span class="error">&#91;422278.843119&#93;</span> ko2iblnd: disagrees about version of symbol ib_get_dma_mr<br/> <span class="error">&#91;422278.843120&#93;</span> ko2iblnd: Unknown symbol ib_get_dma_mr (err -22)<br/> <span class="error">&#91;422278.843131&#93;</span> ko2iblnd: disagrees about version of symbol ib_alloc_pd<br/> <span class="error">&#91;422278.843132&#93;</span> ko2iblnd: Unknown symbol ib_alloc_pd (err -22)<br/> <span class="error">&#91;422278.843143&#93;</span> ko2iblnd: disagrees about version of symbol rdma_set_reuseaddr<br/> <span class="error">&#91;422278.843144&#93;</span> ko2iblnd: Unknown symbol rdma_set_reuseaddr (err -22)<br/> <span class="error">&#91;422278.843148&#93;</span> ko2iblnd: disagrees about version of symbol rdma_connect<br/> <span class="error">&#91;422278.843149&#93;</span> ko2iblnd: Unknown symbol rdma_connect (err -22)<br/> <span class="error">&#91;422278.843154&#93;</span> ko2iblnd: disagrees about version of symbol ib_modify_qp<br/> <span class="error">&#91;422278.843156&#93;</span> ko2iblnd: Unknown symbol ib_modify_qp (err -22)<br/> <span class="error">&#91;422278.843168&#93;</span> ko2iblnd: disagrees about version of symbol rdma_destroy_id<br/> <span class="error">&#91;422278.843169&#93;</span> ko2iblnd: Unknown symbol rdma_destroy_id (err -22)<br/> <span class="error">&#91;422278.843174&#93;</span> ko2iblnd: disagrees about version of symbol rdma_accept<br/> <span class="error">&#91;422278.843176&#93;</span> ko2iblnd: Unknown symbol rdma_accept (err -22)<br/> <span class="error">&#91;422278.843189&#93;</span> ko2iblnd: disagrees about version of symbol ib_dealloc_pd<br/> <span class="error">&#91;422278.843190&#93;</span> ko2iblnd: Unknown symbol ib_dealloc_pd (err -22)<br/> <span class="error">&#91;422278.843195&#93;</span> ko2iblnd: disagrees about version of symbol ib_fmr_pool_map_phys<br/> <span class="error">&#91;422278.843196&#93;</span> ko2iblnd: Unknown symbol ib_fmr_pool_map_phys (err -22)<br/> <span class="error">&#91;422278.843452&#93;</span> LNetError: 29038:0:(api-ni.c:1515:lnet_startup_lndnis()) Can't load LND o2ib, module ko2iblnd, rc=256<br/> <span class="error">&#91;422278.853606&#93;</span> LustreError: 29038:0:(events.c:629:ptlrpc_init_portals()) network initialisation failed</p> <p>Could you guys take a look at this issue.</p> Ubuntu 14.04 <br/> 3.13.0-32-generic LU-6083

IB with Ubuntu 14.04 client

Bug Minor Resolved Fixed Nathaniel Clark Wang Shilong patch Tue, 6 Jan 2015 17:12:28 +0000 Fri, 1 Jul 2016 18:48:42 +0000 Fri, 24 Jun 2016 21:48:45 +0000 Lustre 2.9.0 0 9 <p>BTW, we need apply this <a href="http://review.whamcloud.com/#/c/13129/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13129/</a> for master to make normal compiling pass.</p> <p>Bob</p> <p>Could you please review this patch?</p> <p>Thanks</p> <p>Peter</p> <p>suspect the problem comes from using Mellanox infiniband. Is this really necessary? As far as I know Ubuntu has a perfectly fine inkernel IB stack that is configured in by defauit.</p> <p>don't know if it makes any difference but the kernel version you call out, 3.13.0-32-generic, looks a bit obsolete. The current one in Ubuntu 14.04 is 3.13.0-0.43</p> <p>when you say lustre client are you referring to the upstream lustre client that is part of the Ubuntu kernel tree in drivers/staging/lustre, or the lustre client built from the community lustre git tree built on and for Ubuntu?</p> <p>Sorry for incomplete information, I mean for using git tree built with master branch</p> <p>BTW, i see similar problem reported here.<br/> <a href="https://jira.hpdd.intel.com/browse/LU-5597" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-5597</a></p> <p>We really need use Mellanox infiniband, because for Ubuntu built-in IB did<br/> not work for us.</p> <p>Also I tired again, modprobe will also reported following messages:<br/> [ 7528.946399] ko2iblnd: no symbol version for ib_create_cq<br/> [ 7528.946399] ko2iblnd: Unknown symbol ib_create_cq (err -22)<br/> [ 7528.946409] ko2iblnd: no symbol version for rdma_resolve_addr<br/> [ 7528.946410] ko2iblnd: Unknown symbol rdma_resolve_addr (err -22)<br/> [ 7528.946414] ko2iblnd: no symbol version for ib_reg_phys_mr<br/> [ 7528.946415] ko2iblnd: Unknown symbol ib_reg_phys_mr (err -22)<br/> [ 7528.946419] ko2iblnd: no symbol version for ib_create_fmr_pool<br/> [ 7528.946419] ko2iblnd: Unknown symbol ib_create_fmr_pool (err -22)<br/> [ 7528.946426] ko2iblnd: no symbol version for ib_flush_fmr_pool<br/> [ 7528.946427] ko2iblnd: Unknown symbol ib_flush_fmr_pool (err -22)<br/> [ 7528.946439] ko2iblnd: no symbol version for ib_dereg_mr<br/> [ 7528.946440] ko2iblnd: Unknown symbol ib_dereg_mr (err -22)<br/> [ 7528.946443] ko2iblnd: no symbol version for rdma_reject<br/> [ 7528.946444] ko2iblnd: Unknown symbol rdma_reject (err -22)<br/> [ 7528.946448] ko2iblnd: no symbol version for rdma_disconnect<br/> [ 7528.946449] ko2iblnd: Unknown symbol rdma_disconnect (err -22)<br/> [ 7528.946471] ko2iblnd: no symbol version for rdma_resolve_route<br/> [ 7528.946471] ko2iblnd: Unknown symbol rdma_resolve_route (err -22)<br/> [ 7528.946476] ko2iblnd: no symbol version for rdma_bind_addr<br/> [ 7528.946476] ko2iblnd: Unknown symbol rdma_bind_addr (err -22)<br/> [ 7528.946478] ko2iblnd: no symbol version for rdma_create_qp<br/> [ 7528.946479] ko2iblnd: Unknown symbol rdma_create_qp (err -22)<br/> [ 7528.946483] ko2iblnd: no symbol version for ib_destroy_cq<br/> [ 7528.946484] ko2iblnd: Unknown symbol ib_destroy_cq (err -22)<br/> [ 7528.946486] ko2iblnd: no symbol version for rdma_create_id<br/> [ 7528.946487] ko2iblnd: Unknown symbol rdma_create_id (err -22)<br/> [ 7528.946496] ko2iblnd: no symbol version for rdma_listen<br/> [ 7528.946497] ko2iblnd: Unknown symbol rdma_listen (err -22)<br/> [ 7528.946499] ko2iblnd: no symbol version for rdma_destroy_qp<br/> [ 7528.946500] ko2iblnd: Unknown symbol rdma_destroy_qp (err -22)<br/> [ 7528.946504] ko2iblnd: no symbol version for ib_query_device<br/> [ 7528.946505] ko2iblnd: Unknown symbol ib_query_device (err -22)<br/> [ 7528.946507] ko2iblnd: no symbol version for ib_get_dma_mr<br/> [ 7528.946508] ko2iblnd: Unknown symbol ib_get_dma_mr (err -22)<br/> [ 7528.946514] ko2iblnd: no symbol version for ib_alloc_pd<br/> [ 7528.946515] ko2iblnd: Unknown symbol ib_alloc_pd (err -22)<br/> [ 7528.946522] ko2iblnd: no symbol version for rdma_set_reuseaddr<br/> [ 7528.946523] ko2iblnd: Unknown symbol rdma_set_reuseaddr (err -22)<br/> [ 7528.946525] ko2iblnd: no symbol version for rdma_connect<br/> [ 7528.946526] ko2iblnd: Unknown symbol rdma_connect (err -22)<br/> [ 7528.946529] ko2iblnd: no symbol version for ib_modify_qp<br/> [ 7528.946530] ko2iblnd: Unknown symbol ib_modify_qp (err -22)<br/> [ 7528.946537] ko2iblnd: no symbol version for ib_destroy_fmr_pool<br/> [ 7528.946538] ko2iblnd: Unknown symbol ib_destroy_fmr_pool (err -22)<br/> [ 7528.946540] ko2iblnd: no symbol version for rdma_destroy_id<br/> [ 7528.946540] ko2iblnd: Unknown symbol rdma_destroy_id (err -22)<br/> [ 7528.946543] ko2iblnd: no symbol version for rdma_accept<br/> [ 7528.946544] ko2iblnd: Unknown symbol rdma_accept (err -22)<br/> [ 7528.946553] ko2iblnd: no symbol version for ib_dealloc_pd<br/> [ 7528.946553] ko2iblnd: Unknown symbol ib_dealloc_pd (err -22)<br/> [ 7528.946556] ko2iblnd: no symbol version for ib_fmr_pool_map_phys<br/> [ 7528.946557] ko2iblnd: Unknown symbol ib_fmr_pool_map_phys (err -22)<br/> [ 7528.946987] LNetError: 25709:0:(api-ni.c:1515:lnet_startup_lndnis()) Can't load LND o2ib, module ko2iblnd, rc=256<br/> [ 7529.024336] LustreError: 25709:0:(events.c:629:ptlrpc_init_portals()) network initialisation failed</p> <p>With this solutions: <a href="https://github.com/ahlabenadam/lustre_fix.git" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/ahlabenadam/lustre_fix.git</a></p> <p>Now i could load Lustre with IB successfully, Let me test it further.<br/> But i think in the long term considering, we'd better fix this issue for point of Lustre.</p> <p>Best Regards,<br/> Wang Shilong</p> <p>This sounds very similar to <a href="https://jira.whamcloud.com/browse/LU-5597" title="Kernel Module.symvers take precendene instead of MOFED provided" class="issue-link" data-issue-key="LU-5597"><del>LU-5597</del></a></p> <p><b>FYI: Kernel Compatibility</b><br/> Ubuntu 14.04<br/> MLNX 2.4-1.0.4</p> <p>3.13 - Yes<br/> 3.16 - NO<br/> 3.19 - NO</p> <p>Ubuntu 14.04<br/> MLNX 3.0-2.0.1</p> <p>3.13 - ?<br/> 3.16 - Yes<br/> 3.19 - ?</p> <p>Modified Build Instructions without need for <a href="https://github.com/ahlabenadam/lustre_fix.git" class="external-link" target="_blank" rel="nofollow noopener">ahlabenadam fix</a>:</p> <p>Install MLNX OFED as normal (must use 3.13 kernel for MLNX 2.4-1.0.4)<br/> As <em>root</em>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cd /usr/src/ofa_kernel ./ofed_scripts/gen-compat-autoconf.sh include/linux/compat-3.13.h &gt; include/linux/compat_autoconf.h export MODULES_DIR=/lib/modules/$(uname -r)/updates/dkms/./ ./ofed_scripts/create_Module.symvers.sh </pre> </div></div> <p>In <tt>lustre-release</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./configure --with-o2ib=/usr/src/ofa_kernel --disable-server --enable-quota </pre> </div></div> <p><b>NOTE:</b> compilation problem still exists <a href="https://jira.whamcloud.com/browse/LU-5628" title="Dealing with kernels that have lustre enabled already" class="issue-link" data-issue-key="LU-5628"><del>LU-5628</del></a>, can be partially solved by adding <tt>--with-max-payload-mb=1</tt> to configure line and then editing config.h to replace <tt>((1)&lt;&lt;20)</tt> with <tt>1048576</tt></p> <p>What is left for this work besides adding Documentation? For me everything works well.</p> <p>This issue is handled by patch <a href="http://review.whamcloud.com/20523" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20523</a> linked to <a href="https://jira.whamcloud.com/browse/LU-5953" title="lustre[-dkms] needs to automatically account for OFED" class="issue-link" data-issue-key="LU-5953"><del>LU-5953</del></a></p> <p>Patch landed to master</p> Related LU-1706 LU-5953 LU-5628 Development End date Wed, 1 Jun 2016 17:12:28 +0000 Rank 1|hzx393: Rank (Obsolete) 16928 Severity Start date Tue, 6 Jan 2015 17:12:28 +0000