https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5444 Lustre <p>Dear Team,</p> <p>In our setup we integrating lustre with intel phi.<br/> After following steps, we are able to build the lustre mic rpm.<br/> But at the time of installation we are getting error.</p> <p>rpm -ivh /root/phi/mpss-3.3/mpss-sdk-k1om-3.3-1.x86_64.rpm<br/> tar -xvjf /root/phi/mpss-3.3/src/linux-2.6.38+mpss3.3.tar.bz2<br/> rpm2cpio mpss-3.3/k1om/kernel-dev-2.6.38+mpss3.3-1.knightscorner.rpm | cpio -idmv<br/> cp boot/Module.symvers-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/Module.symvers<br/> cp boot/config-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/.config<br/> make ARCH=k1om silentoldconfig modules_prepare<br/> source /opt/mpss/3.3/environment-setup-k1om-mpss-linux<br/> export LD=k1om-mpss-linux-ld<br/> ./configure --with-linux=/root/phi/linux-2.6.38+mpss3.3 --disable-server --disable-tests --disable-doc --with-o2ib=/usr/src/ofa_kernel/default --host=k1om-mpss-linux --build=x86_64-pc-linux<br/> Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.x86_64<br/> Wrote: /root/rpmbuild/SRPMS/lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.src.rpm<br/> Wrote: /root/rpmbuild/RPMS/x86_64/lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> Wrote: /root/rpmbuild/RPMS/x86_64/lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> Wrote: /root/rpmbuild/RPMS/x86_64/lustre-client-mic-source-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> Wrote: /root/rpmbuild/RPMS/x86_64/lustre-iokit-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> Wrote: /root/rpmbuild/RPMS/x86_64/lustre-client-mic-debuginfo-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> Error<br/> <span class="error">&#91;root@phi2 ~&#93;</span># rpm -ivh /root/rpmbuild/RPMS/x86_64/lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm<br/> error: Failed dependencies:<br/> kernel = 2.6.38.8+mpss3.3 is needed by lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64<br/> <span class="error">&#91;root@phi2 ~&#93;</span>#</p> <p>Need your help in proceeding further..</p> <p>Thank You<br/> Atul Yadav</p> MPSS 3.3<br/> Lustre 2.5.2 LU-5444

Lustre For MPSS3.3

Bug Critical Resolved Fixed Dmitry Eremin Atul Yadav Fri, 1 Aug 2014 19:47:31 +0000 Wed, 20 Aug 2014 20:04:07 +0000 Wed, 20 Aug 2014 20:04:06 +0000 Lustre 2.5.2 0 4 <p>Hello Team,<br/> Any Update on this requirement.</p> <p>You should do one more step before configure and change configure arguments like bellow:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ rpm2cpio /root/phi/mpss-3.3/ofed/modules/ofed-driver-devel-2.6.32-220.el6.x86_64-3.3-1.x86_64.rpm | cpio -idm $ ./configure --with-linux=/root/phi/linux-2.6.38+mpss3.3 --disable-server --disable-tests --disable-doc \ --with-o2ib=/root/phi/usr/src/ofed-driver --host=k1om-mpss-linux --build=x86_64-pc-linux </pre> </div></div> <p>The issue with incorrect dependency you can avoid by adding <tt>--nodeps</tt> option to rpm.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rpm -ivh --nodeps /root/rpmbuild/RPMS/x86_64/lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm </pre> </div></div> <p>It's a known issue and will be fixed in next release.</p> <p>Dear team,</p> <p>As per the input received from your side, we are able to build lustre mic modules without any problem.<br/> But at the time of lustre mounting we are getting error.<br/> Sharing full details with you.<br/> yum install libselinux-devel -y<br/> rpm -ivh /root/phi/mpss-3.3/modules/mpss-modules-dev-2.6.32-431.el6.x86_64-3.3-1.x86_64.rpm<br/> rpm -ivh /root/phi/mpss-3.3/modules/mpss-modules-2.6.32-431.el6.x86_64-3.3-1.x86_64.rpm<br/> rpm -Uvh mpss-3.3/ofed/ofed-ibpd-3.3-r0.glibc2.12.2.x86_64.rpm<br/> rpmbuild --rebuild --define "MOFED 1" mpss-3.3/src/dapl-2.0.42.2-1.glibc2.12.2.src.rpm mpss-3.3/src/libibscif-1.0.0-1.fc13.src.rpm mpss-3.3/src/ofed-driver-3.3-1.src.rpm<br/> rpm -ivh /root/phi/mpss-3.3/mpss-sdk-k1om-3.3-1.x86_64.rpm<br/> tar -xvjf /root/phi/mpss-3.3/src/linux-2.6.38+mpss3.3.tar.bz2<br/> rpm2cpio mpss-3.3/k1om/kernel-dev-2.6.38+mpss3.3-1.knightscorner.rpm | cpio -idmv<br/> rpm2cpio /root/phi/mpss-3.3/ofed/modules/ofed-driver-devel-2.6.32-220.el6.x86_64-3.3-1.x86_64.rpm | cpio -idmv<br/> cp boot/Module.symvers-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/Module.symvers<br/> cp boot/config-2.6.38.8+mpss3.3 linux-2.6.38+mpss3.3/.config<br/> cd linux-2.6.38+mpss3.3/<br/> make ARCH=k1om silentoldconfig modules_prepare<br/> source /opt/mpss/3.3/environment-setup-k1om-mpss-linux<br/> export LD=k1om-mpss-linux-ld<br/> cd /root/phi/lustre-release<br/> git checkout 2.5.2<br/> sh autogen.sh<br/> ./configure --with-linux=/root/phi/linux-2.6.38+mpss3.3 --disable-server --disable-tests --disable-doc --with-o2ib=/usr/src/ofed-driver --host=k1om-mpss-linux --build=x86_64-pc-linux<br/> rpm -ivvh /root/rpmbuild/RPMS/x86_64/lustre-client-mic-modules-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm --nodeps<br/> rpm -ivh /root/rpmbuild/RPMS/x86_64/lustre-client-mic-2.5.2-2.6.38.8+mpss3.3.x86_64.rpm --nodeps</p> <p>mic0# modprobe -v lnet<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/net/lustre/libcfs.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/net/lustre/lnet.ko networks=o2ib0(ib0),o2ib1(ib1) forwarding="enabled"<br/> mic0#modprobe -v lustre<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/lvfs.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/obdclass.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/ptlrpc.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/fid.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/mdc.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/osc.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/lov.ko<br/> insmod /lib/modules/2.6.38.8+mpss3.3/extra/kernel/fs/lustre/lustre.ko</p> <p>mic0# mount.lustre 192.168.3.101@o2ib:/lustre /lustre</p> <p>Error On MGS node<br/> Aug 5 23:59:32 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib<br/> Aug 5 23:59:57 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib<br/> Aug 6 00:00:22 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2267:kiblnd_passive_connect()) Can't accept 192.168.2.88@o2ib on 192.168.2.101@o2ib (ib1:1:192.168.3.101): bad dst nid 192.168.3.101@o2ib<br/> Aug 6 00:11:01 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) 192.168.2.88@o2ib: REJECTED 28<br/> Aug 6 00:11:26 IO1 kernel: LNetError: 2139:0:(o2iblnd_cb.c:2923:kiblnd_cm_callback()) 192.168.3.88@o2ib1: REJECTED 28</p> <p>MDS <br/> <span class="error">&#91;root@IO1 ~&#93;</span># lctl list_nids<br/> 192.168.2.101@o2ib<br/> 192.168.3.101@o2ib1</p> <p>MIC0#<br/> 192.168.2.88@o2ib<br/> 192.168.3.88@o2ib1</p> <p>need assistance to resolve the errors </p> <p>thank You<br/> Atul Yadav</p> <p>Most probably you should swap IP address of ib0 and ib1. Try to reach servers from MIC card by just <tt>ping 192.168.2.101</tt> and <tt>ping 192.168.3.101</tt> and be sure it works. Then make sure that name of IB interface the same as on server. For example, if the name o2ib0 correspond to 192.168.2.88 on MIC the same name o2ib0 should correspond to 192.168.2.101 on server.</p> <p>Dear Team,</p> <p>After swapping the IP address also it's not working .</p> <p>We are able to ping the IB address, but we are not able to ping through lctl.</p> <p>Thank You <br/> Atul Yadav</p> <p>According your logs the Lustre client works but Lustre configuration is incorrect. Please, check /etc/modprobe.d/lustre.conf on servers and clients for correct configuration. It seems you have an issue with gateway configuration between two networks.</p> <p>Maybe you just need to change the mount command to following:<br/> <tt># mount.lustre 192.168.3.101@o2ib<font color="red">1</font>:/lustre /lustre</tt></p> <p>Just for info: I updated instructions in my <a href="https://wiki.hpdd.intel.com/display/~dmiter/Lustre+on+Intel+Xeon+Phi" class="external-link" target="_blank" rel="nofollow noopener">WiKi page</a>. It contains detailed steps how to build Lustre with MPSS 3.3.</p> Development Epic/Theme Lustre-2.5.2 MPSS-3.3 Rank 1|hzwsun: Rank (Obsolete) 15160 Severity