https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5348 Lustre <p>Dear Team,</p> <p>In our HPC enviorment, we are using 2 sets of computational servers.<br/> 1: CPU Group<br/> 2: Phi Group</p> <p>For both the groups we are using lustre-2.5.2.tar.gz, but different OFED stack.<br/> For CPU Group:-<br/> lustre-2.5.2.tar.gz+ MLNX_OFED_LINUX-2.2-1.0.1-rhel6.5-x86_64.tgz <br/> <span class="error">&#91;root@test1 ~&#93;</span># cat /etc/redhat-release<br/> CentOS release 6.5 (Final)<br/> <span class="error">&#91;root@test1 ~&#93;</span># uname -r<br/> 2.6.32-431.el6.x86_64<br/> <span class="error">&#91;root@test1 ~&#93;</span># rpm -qa | grep kernel<br/> kmod-mlnx-ofa_kernel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64<br/> libreport-plugin-kerneloops-2.0.9-19.el6.centos.x86_64<br/> dracut-kernel-004-335.el6.noarch<br/> kernel-firmware-2.6.32-431.el6.noarch<br/> kernel-devel-2.6.32-431.el6.x86_64<br/> kernel-headers-2.6.32-431.el6.x86_64<br/> mlnx-ofa_kernel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64<br/> mlnx-ofa_kernel-devel-2.2-OFED.2.2.1.0.0.1.gdf6fefb.rhel6u5.x86_64<br/> abrt-addon-kerneloops-2.0.8-21.el6.centos.x86_64<br/> kernel-2.6.32-431.el6.x86_64<br/> kmod-kernel-mft-mlnx-3.6.0-24.rhel6u5.x86_64<br/> <span class="error">&#91;root@test1 ~&#93;</span># rpm -qa | grep lustre<br/> lustre-client-modules-2.5.2-2.6.32_431.el6.x86_64.x86_64<br/> lustre-client-2.5.2-2.6.32_431.el6.x86_64.x86_64<br/> <span class="error">&#91;root@test1 ~&#93;</span># ibv_devices<br/> device node GUID<br/> ------ ----------------<br/> mlx4_0 f452140300832610<br/> <span class="error">&#91;root@test1 opt&#93;</span># modprobe -v lnet<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/libcfs.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/lnet.ko networks=o2ib(ib0)<br/> <span class="error">&#91;root@test1 opt&#93;</span># modprobe -v lustre<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lvfs.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/obdclass.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/ptlrpc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/fid.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/mdc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/osc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lov.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lustre.ko</p> <p>For Phi node group:-<br/> lustre-2.5.2.tar.gz+ OFED-1.5.4.1.tgz+ OFED-3.5-2-MIC-beta1.tgz<br/> <span class="error">&#91;root@phi2 ~&#93;</span># cat /etc/redhat-release<br/> CentOS release 6.5 (Final)<br/> <span class="error">&#91;root@phi2 ~&#93;</span># uname -r<br/> 2.6.32-431.el6.x86_64<br/> <span class="error">&#91;root@phi2 ~&#93;</span># rpm -qa | grep kernel<br/> libreport-plugin-kerneloops-2.0.9-19.el6.centos.x86_64<br/> dracut-kernel-004-335.el6.noarch<br/> kernel-firmware-2.6.32-431.el6.noarch<br/> kernel-devel-2.6.32-431.el6.x86_64<br/> kernel-headers-2.6.32-431.el6.x86_64<br/> abrt-addon-kerneloops-2.0.8-21.el6.centos.x86_64<br/> kernel-2.6.32-431.el6.x86_64<br/> glibc2.12.2pkg-mpss-rasmm-kernel-3.2.3-1.glibc2.12.2.x86_64<br/> <span class="error">&#91;root@phi2 ~&#93;</span># rpm -qa | grep lustre<br/> lustre-client-2.5.2-2.6.32_431.el6.x86_64.x86_64<br/> lustre-client-modules-2.5.2-2.6.32_431.el6.x86_64.x86_64<br/> <span class="error">&#91;root@phi2 ~&#93;</span># ibv_devices<br/> device node GUID<br/> ------ ----------------<br/> scif0 4c79bafffe540097<br/> mlx4_0 f45214030082d930<br/> <span class="error">&#91;root@phi2 ~&#93;</span># modprobe -v lnet<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/libcfs.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/net/lustre/lnet.ko networks=o2ib(ib0)<br/> <span class="error">&#91;root@phi2 ~&#93;</span># modprobe -v lustre<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lvfs.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/obdclass.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/ptlrpc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/fid.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/mdc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/osc.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lov.ko<br/> insmod /lib/modules/2.6.32-431.el6.x86_64/extra/kernel/fs/lustre/lustre.ko</p> <p>First Query:<br/> Can we use 2 different OFED stack in single lustre file system.</p> <p>Second Query:<br/> We are using plain kernel in our setup. Is it advisable.</p> <p>Please validate our setup and reply us.</p> <p>Thank You<br/> Atul Yadav</p> CentOS release 6.5 (Final)<br/> lustre-2.5.2.tar.gz+ OFED-1.5.4.1.tgz+ OFED-3.5-2-MIC-beta1.tgz<br/> lustre-2.5.2.tar.gz+ MLNX_OFED_LINUX-2.2-1.0.1-rhel6.5-x86_64.tgz LU-5348

OFED Working

Question/Request Major Resolved Fixed Dmitry Eremin Atul Yadav Tue, 15 Jul 2014 04:08:08 +0000 Tue, 15 Jul 2014 17:18:04 +0000 Tue, 15 Jul 2014 17:18:04 +0000 0 4 <p>The OFED stack from Xeon Phi depends and closely communicate with Xeon host OFED stack. So, You cannot use both stacks simultaneously. But you can switch them during reboot or modules load/unload.</p> <p>Dear Sir,</p> <p>In current setup, we are using 3 type of OFED stack.</p> <p>1 IO Node(MDS+ OSS) Cent OS 6.5 Native IB support <br/> 2 Compute Node Mellanox IB Driver<br/> 3 Phi Compute node OFED Driver</p> <p>Is this setup is ok ?<br/> Can we use os native IB in IO nodes and in compute nodes with mellanox.? <br/> Is it required to maintain same OFED driver in all the environment ? </p> <p>What are the basic IB modules are required by lustre to work without any problem.</p> <p>Mellanox Module loaded in OS<br/> rdma_ucm<br/> rdma_cm<br/> ib_addr<br/> ib_ipoib<br/> mlx4_core<br/> mlx4_ib<br/> mlx4_en<br/> mlx5_core<br/> mlx5_ib<br/> ib_uverbs<br/> ib_umad<br/> ib_ucm<br/> ib_sa<br/> ib_cm<br/> ib_mad<br/> ib_core</p> <p>Ofed 1.5.4.1 + OFED-3.5-2-MIC-beta1 loaded in OS<br/> rdma_ucm<br/> rdma_cm<br/> ib_addr<br/> ib_ipoib<br/> mlx4_core<br/> mlx4_ib<br/> mlx4_en<br/> mlx5_core<br/> mlx5_ib<br/> ib_mthca<br/> ib_uverbs<br/> ib_umad<br/> ib_sa<br/> ib_cm<br/> ib_mad<br/> ib_core<br/> iw_cxgb3<br/> iw_cxgb4<br/> iw_nes</p> <p>Please reply back</p> <p>Thank You<br/> Atul Yadav</p> <p>Not sure I got the question right. Let's try to describe behavior. Lustre uses Kernel VERB API to communicate through IB. This ability provided by kernel loadable modules from kernel-ib package. It's not so critical what particular version of those drivers installed if they works. The important things is the version of drivers installed should correspond the version of drivers which Lustre build with. On different nodes this combination can be different. User space libraries from OFED stack are not used by Lustre at all.</p> <p>Thanks for response,</p> <p>I got my answer through your reply.</p> <p>Question answered.</p> Development Epic/Theme Lustre-2.5.2 Mellaonx OFED Rank 1|hzwrfz: Rank (Obsolete) 14914 Story Points 11.0