[LU-5348] OFED Working

• Type: Question/Request
• Resolution: Fixed
• Priority: Major
• Environment: CentOS release 6.5 (Final); lustre-2.5.2.tar.gz+ OFED-1.5.4.1.tgz+ OFED-3.5-2-MIC-beta1.tgz; lustre-2.5.2.tar.gz+ MLNX_OFED_LINUX-2.2-1.0.1-rhel6.5-x86_64.tgz

Epic/Theme

Lustre-2.5.2 Mellaonx OFED

• Story Points: 11
• Assignee: Dmitry Eremin (Inactive)
• Reporter: Atul Yadav

Description

Dear Team,

In our HPC enviorment, we are using 2 sets of computational servers.
1: CPU Group
2: Phi Group

For both the groups we are using lustre-2.5.2.tar.gz, but different OFED stack.
For CPU Group:-
lustre-2.5.2.tar.gz+ MLNX_OFED_LINUX-2.2-1.0.1-rhel6.5-x86_64.tgz
[root@test1 ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@test1 ~]# uname -r
2.6.32-431.el6.x86_64
[root@test1 ~]# rpm -qa | grep kernel
kmod-mlnx-ofa_kernel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64
libreport-plugin-kerneloops-2.0.9-19.el6.centos.x86_64
dracut-kernel-004-335.el6.noarch
kernel-firmware-2.6.32-431.el6.noarch
kernel-devel-2.6.32-431.el6.x86_64
kernel-headers-2.6.32-431.el6.x86_64
mlnx-ofa_kernel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64
mlnx-ofa_kernel-devel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64
abrt-addon-kerneloops-2.0.8-21.el6.centos.x86_64
kernel-2.6.32-431.el6.x86_64
kmod-kernel-mft-mlnx-3.6.0-24.rhel6u5.x86_64
[root@test1 ~]# rpm -qa | grep lustre
lustre-client-modules-2.5.2-2.6.32_431.el6.x86_64.x86_64
lustre-client-2.5.2-2.6.32_431.el6.x86_64.x86_64
[root@test1 ~]# ibv_devices
device node GUID
------ ----------------
mlx4_0 f452140300832610
[root@test1 opt]# modprobe -v lnet
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/libcfs.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/lnet.ko networks=o2ib(ib0)
[root@test1 opt]# modprobe -v lustre
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lvfs.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/obdclass.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/ptlrpc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/fid.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/mdc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/osc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lov.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lustre.ko

For Phi node group:-
lustre-2.5.2.tar.gz+ OFED-1.5.4.1.tgz+ OFED-3.5-2-MIC-beta1.tgz
[root@phi2 ~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
[root@phi2 ~]# uname -r
2.6.32-431.el6.x86_64
[root@phi2 ~]# rpm -qa | grep kernel
libreport-plugin-kerneloops-2.0.9-19.el6.centos.x86_64
dracut-kernel-004-335.el6.noarch
kernel-firmware-2.6.32-431.el6.noarch
kernel-devel-2.6.32-431.el6.x86_64
kernel-headers-2.6.32-431.el6.x86_64
abrt-addon-kerneloops-2.0.8-21.el6.centos.x86_64
kernel-2.6.32-431.el6.x86_64
glibc2.12.2pkg-mpss-rasmm-kernel-3.2.3-1.glibc2.12.2.x86_64
[root@phi2 ~]# rpm -qa | grep lustre
lustre-client-2.5.2-2.6.32_431.el6.x86_64.x86_64
lustre-client-modules-2.5.2-2.6.32_431.el6.x86_64.x86_64
[root@phi2 ~]# ibv_devices
device node GUID
------ ----------------
scif0 4c79bafffe540097
mlx4_0 f45214030082d930
[root@phi2 ~]# modprobe -v lnet
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/libcfs.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/lnet.ko networks=o2ib(ib0)
[root@phi2 ~]# modprobe -v lustre
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lvfs.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/obdclass.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/ptlrpc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/fid.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/mdc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/osc.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lov.ko
insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lustre.ko

First Query:
Can we use 2 different OFED stack in single lustre file system.

Second Query:
We are using plain kernel in our setup. Is it advisable.

Please validate our setup and reply us.

Thank You
Atul Yadav