https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2614 Lustre <p>Change lbuild script to support compilation of Mellanox (alpha) OFED stack<br/> to support SR-IOV virtualisation for IB HCAs</p> <p>NOTE: Work needed to support the testing of IB Virtualization on the Julich cluster,</p> LU-2614

Add build support for HCA virtualisation

Story Critical Resolved Not a Bug Frank Heckes Frank Heckes Mon, 14 Jan 2013 08:59:14 +0000 Fri, 1 Mar 2019 22:41:27 +0000 Fri, 1 Mar 2019 22:41:27 +0000 Lustre 2.4.0 0 6 <p>Out of curiosity is this their OFED-1.9 stack?</p> <p>No, not as far as I know. We working with the pre-release version supporting SR-IOV for <br/> Mellanox HCA's that is available for RHEL/CentOS 6.2 only, just to prepare the<br/> build system and autotest framework to make use of the feature quickly for 'production' <br/> as soon as support for 6.3 and other distros will be released.</p> <p>Frank, any reason why this is for 2.2 and not 2.4?</p> <p>At the moment the Mellanox SR-IOV enabled OFED only<br/> supports kernel version 2.6.32-220.X. (Tried to do for current<br/> master branch, but naive build failed and I would end up in coding the ofa kernel)<br/> This kernel seems to be available in b2_2 only (please correct me if I'm wrong).<br/> I expect Rhel/CentOS6.3 supported OFED to be shared with us<br/> next Monday.<br/> Anyway, I decided to use this branch to create a prototype. Looks<br/> very promissing all build succeed (i.e Lustre was built against<br/> the SR-IOV OFED) and we're going to run autotest against it today.<br/> Once that work, we can easily replace the new OFED to run build of the <br/> master branch.</p> <p>Changed lbuild to compile SR-IOV OFED stack and compile Lustre against it.<br/> Software builds executed successful (see: <a href="http://build.whamcloud.com/job/lustre-reviews-vib/arch=x86_64,build_type=server,distro=el6,ib_stack=vofa/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews-vib/arch=x86_64,build_type=server,distro=el6,ib_stack=vofa/</a>)<br/> Constraints: Rhel/CentOS 6.2, kernel 2.6.32.220.* for server<br/> Autotest ran successful, too. (although test_123a failed; see: <a href="https://maloo.whamcloud.com/test_sets/c20ab082-6633-11e2-a42b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c20ab082-6633-11e2-a42b-52540035b04c</a> , it proves the concept) </p> <p>Wait for Rhel/CentOS 6.3 OFED stack to use the framework changes (jenkins, lbuild) for master branch and finally land the change.</p> <p>Received Mellanox (SR-IOV enabled) OFED stack for CentOS 6.3.<br/> Made changes to compile Lustre against Mellanox OFED stack (see gerrit:5295 patch 17)<br/> Build has been started in test queue 'Frank_Heckes'; waiting for test results.</p> <p>Build tested almost successful, besides test_0c, test_0d and test_73a of 'replay-single' suite. These error are related to bug <a href="https://jira.whamcloud.com/browse/LU-414" title="error looking up logfile" class="issue-link" data-issue-key="LU-414"><del>LU-414</del></a>.<br/> Installation and all other tests ran successful which proves the concept. <br/> See session :</p> <p><a href="https://maloo.whamcloud.com/test_sessions/0c32a460-a0dd-11e2-b429-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/0c32a460-a0dd-11e2-b429-52540035b04c</a></p> <p>The build is currently available on Whamcloud internal jenkins-node 'jenkins-staging'. I'll make them available asap.</p> <p><a href="http://review.whamcloud.com/#change,5295" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5295</a></p> <p>Pushed change that will change the RPM specfile to build the Mellanox kernel modules and kernel-devel RPM to fit into the Jenkins build environment:</p> <p><a href="http://review.whamcloud.com/#/c/11593/2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11593/2</a></p> <p>The change have been successful tested on node identical to onyx Jenkins builder on Jülich autotest cluster Juliette.<br/> The RPMs have been installed, a Lustre FS had been created and tested successful using on single SR-IOV enable node of Juliette.</p> <p>Anyway change is too inflexible. Will supply new change which will only change the --o2ib-path flag for Lustre configure.</p> <ul> <li>New patch relying on the modification of variable CONFIGURE_FLAGS only to provide the correct<br/> path of the Mellanox OFED compat header files to Lustre configure (via <tt>--with-o2ib=...</tt>) has been<br/> pushed:<br/> <a href="http://review.whamcloud.com/#/c/11593/3" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11593/3</a><br/> Build ended successful, but code need a correction and additionally an extension to download the Mellanox OFED tar archive<br/> automatically.</li> <li>Change need to be test also in Jenkins job 'lustre-reviews-vib', since configuration of 'lustre-reviews'<br/> won't cover the execution of MLNX builds (vib or mlnx label + setting in script file is missing in lustre-reviews)<br/> Queue has to be enable as soon as patch is review +1 in lustre-reviews.<br/> Code for Mellanox build has been verified on community cluster Juliette (FZ Jülich) so far only.</li> <li>Checked the execution of normal Lustre 'inkernel' build in VM that run on top of a hypervisor with Mellanox OFED installed<br/> and SR-IOV enabled. It pass all basic test (creation of FS, client file operations (create, remove, ...), small performance test)<br/> This would increase the test coverage for 'inkernel' and if available openfabric builds, too. Anyway some technical aspects (does<br/> all user land libs and programs execute well without runtime surprises; especially for all MPI applications) has to clarified <br/> and discussed.</li> </ul> <p>closing due to we have IB support in our lbuild for a while now</p> Related Development Rank 1|hzvf9b: Rank (Obsolete) 6117