[LU-8785] Setting up lustre client on ppc64le-ubuntu 16.04 ( having lustre built from source )

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Blocker
• Reporter: Amitkumar Ghatwal
• Assignee: WC Triage
• Resolution: Fixed

Environment

root@pts00433-vm22:~/lustre-release# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"

root@pts00433-vm22:~/lustre-release# lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 3
On-line CPU(s) list: 0-2
Thread(s) per core: 1
Core(s) per socket: 3
Socket(s): 1
NUMA node(s): 1
Model: 2.1 (pvr 004b 0201)
Model name: POWER8E (raw), altivec supported
Hypervisor vendor: KVM
Virtualization type: para
L1d cache: 64K
L1i cache: 32K
NUMA node0 CPU(s): 0-2

• Epic/Theme: Lustre-2.8.0
• Story Points: 1
• Business Value: 1
• Severity: 3

Description

Hi All,

I have a working 3 node cluster set on my x86_64 machines ( centos ) and i have followed the steps as per this url : https://gist.github.com/joshuar/4e283308c932ec62fc05 as shown below

1. Machine 1 : Acting as Management server( MGS) , Management target ( MGT ) ,Metadata server( MDS) , Metadata target ( MDT ) - CentOS6.6 x86-VM1 - 10.51.225.95
2. Machine 2 : Acting as - Object Storage Server ( OSS ) , Object Storage Target ( OST ) CentOS6.8 x86-VM2 - 10.51.225.96
3. Machine 3 : Acting as lustre client - CentOS 7.2.1511- VM3 - 10.51.231.200

All above machines are x86_64.

Machine 1 , is showing Linux kernel patched with lustre : 2.6.32-573.12.1.el6_lustre.x86_64
Machine 2 , is showing Linux kernel patched with lustre : 2.6.32-573.12.1.el6_lustre.x86_64
Machine 3 : is showing the normal Linux kernel : 3.10.0-327.10.1.el7.x86_64

And to verify that lustre client is working on machine 3 , i have done the below on machine 3 .

The lfs df -h command lists space usage per OST and the MDT in human-readable format. This command generates below output on lustre client :

On Client Node - Machine 3
[root@hj-ibmibm2026 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
whatevs-MDT0000_UUID 28.4G 58.9M 26.3G 0% /mnt/lustre[MDT:0]
whatevs-OST0000_UUID 39.5G 49.0M 37.3G 0% /mnt/lustre[OST:0]

filesystem summary: 39.5G 49.0M 37.3G 0% /mnt/lustre

Now i need to replace the machine 3 (x86_64 ) with machine having ppc64le architecture . So i need to have lustre client on ppc64le machines ( ubuntu and red hat version ) . So i went to build lustre from source using below steps.

root@pts00433-vm22:~# git clone git://git.hpdd.intel.com/fs/lustre-release.git
root@pts00433-vm22:~# cd lustre-release/
root@pts00433-vm22:~/lustre-release# sh ./autogen.sh
configure.ac:15: installing 'config/compile'
configure.ac:10: installing 'config/config.guess'
configure.ac:10: installing 'config/config.sub'
configure.ac:12: installing 'config/install-sh'
configure.ac:12: installing 'config/missing'
libcfs/libcfs/autoMakefile.am:39: warning: source file 'util/string.c' is in a subdirectory,
libcfs/libcfs/autoMakefile.am:39: but option 'subdir-objects' is disabled
automake: warning: possible forward-incompatibility.
automake: At least a source file is in a subdirectory, but the 'subdir-objects'
........
........
........
lustre/utils/Makefile.am:28: but option 'subdir-objects' is disabled
lustre/utils/Makefile.am:24: warning: source file 'gss/sk_utils.c' is in a subdirectory,
lustre/utils/Makefile.am:24: but option 'subdir-objects' is disabled
root@pts00433-vm22:~/lustre-release#./configure --disable-server

....
LD: /usr/bin/ld
CPPFLAGS: -include /root/lustre-release/undef.h -include /root/lustre-release/config.h -I/root/lustre-release/libcfs/include -I/root/lustre-release/lnet/include -I/root/lustre-release/lustre/include
CFLAGS: -g -O2
EXTRA_KCFLAGS: -include /root/lustre-release/undef.h -include /root/lustre-release/config.h -g -I/root/lustre-release/libcfs/include -I/root/lustre-release/lnet/include -I/root/lustre-release/lustre/include

Type 'make' to build Lustre.
root@pts00433-vm22:~/lustre-release# make
make all-recursive
make[1]: Entering directory '/root/lustre-release'
Making all in .
make[2]: Entering directory '/root/lustre-release'
grep -v config.h.in config.h.in > undef.h
make LDFLAGS= CC="gcc" -C /lib/modules/4.4.0-31-generic/build \
-f /root/lustre-release/build/Makefile LUSTRE_LINUX_CONFIG=/lib/modules/4.4.0-31-generic/build/.config \
LINUXINCLUDE='-I$(srctree)/arch/$(SRCARCH)/include -Iarch/$(SRCARCH)/include/generated -Iinclude $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) -I$(srctree)/arch/$(SRCARCH)/include/uapi -Iarch/$(SRCARCH)/include/generated/uapi -I$(srctree)/include/uapi -Iinclude/generated/uapi -include /lib/modules/4.4.0-31-generic/build/include/linux/kconfig.h' \
M=/root/lustre-release -o tmp_include_depends -o scripts -o \
include/config/MARKER modules
make[3]: Entering directory '/usr/src/linux-headers-4.4.0-31-generic'
LD /root/lustre-release/built-in.o
LD /root/lustre-release/libcfs/built-in.o
LD /root/lustre-release/libcfs/libcfs/built-in.o
.....
.....
....
make[3]: Leaving directory '/root/lustre-release/lustre/osc'
make[3]: Entering directory '/root/lustre-release/lustre'
make[3]: Nothing to be done for 'all-am'.
make[3]: Leaving directory '/root/lustre-release/lustre'
make[2]: Leaving directory '/root/lustre-release/lustre'
make[1]: Leaving directory '/root/lustre-release'
root@pts00433-vm22:~/lustre-release# ./LUSTRE-VERSION-GEN
2.8.59_78_g7d4106c
root@pts00433-vm22:~/lustre-release# whereis lustre
lustre: /usr/lib/lustre /usr/include/lustre /usr/share/lustre /usr/share/man/man7/lustre.7
root@pts00433-vm22:~/lustre-release# make install
make[3]: Leaving directory '/root/lustre-release/lustre/osc'
make[2]: Leaving directory '/root/lustre-release/lustre/osc'
make[2]: Entering directory '/root/lustre-release/lustre'
make[3]: Entering directory '/root/lustre-release/lustre'
make[3]: Nothing to be done for 'install-exec-am'.
make[3]: Nothing to be done for 'install-data-am'.
make[3]: Leaving directory '/root/lustre-release/lustre'
make[2]: Leaving directory '/root/lustre-release/lustre'
make[1]: Leaving directory '/root/lustre-release/lustre'
root@pts00433-vm22:~/lustre-release#

So now assuming lustre has been installed from source correctly on this ppc64le-ubuntu16.04 vm , need to configure this machine to act as a lustre client.

Getting issue while configuring the lustre client with below steps
On the client:
Load the lustre kernel module using modprobe lustre.
root@pts00433-vm22:~/lustre-release# modprobe lustre
modprobe: FATAL: Module lustre not found in directory /lib/modules/4.4.0-31-generic

Please do let me know having built lustre from source successfully ( at least that's what i think from above steps ) how to configure this ubuntu 16.04 machine as "lustre client" so that i replace the above foremost mentioned x86_64 machine with this ppc64le machine as lustre client

Any pointers will be appreciated.

Regards
Amit

Comments

[James A Simmons]

Here too I think "depmod -a" will resolve this.

[Amitkumar Ghatwal]

Yes , that is what i did yesterday and got it working, but replying it here now
Next below commands was how i managed to install lustre client

1. depmod -a
2. modprobe -v lustre

Installed at power up
[root@pts00433-vm5 ~]# cat /etc/sysconfig/modules/lustre.module
#!/bin/sh

/sbin/lsmod | /bin/grep lustre 1>/dev/null 2>&1
if [ ! $? ] ; then
/sbin/modprobe lustre >/dev/null 2>&1
fi

connect to MGS/OSS

1. mkdir /mnt/first_mount
2. mount -t lustre 10.51.225.95@tcp0:/whatevs /mnt/first_mount

Confirming that the client is configured correctly
#[root@pts00433-vm5 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
whatevs-MDT0000_UUID 28.4G 58.9M 26.3G 0% /mnt/first_mount[MDT:0]
whatevs-OST0000_UUID 39.5G 57.0M 37.3G 0% /mnt/first_mount[OST:0]

filesystem summary: 39.5G 57.0M 37.3G 0% /mnt/first_mount

1. [root@pts00433-vm5 ~]# df -h
   Filesystem Size Used Avail Use% Mounted on
   /dev/mapper/rhel-root 115G 88G 28G 77% /
   devtmpfs 7.5G 0 7.5G 0% /dev
   tmpfs 7.5G 0 7.5G 0% /dev/shm
   tmpfs 7.5G 137M 7.4G 2% /run
   tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
   /dev/vda2 497M 310M 187M 63% /boot
   10.51.225.95@tcp:/whatevs 40G 58M 38G 1% /mnt/first_mount
   tmpfs 1.5G 0 1.5G 0% /run/user/0

The above tells me that machine 3 is successfully configured to be working as lustre client . Do let me know of any more check/test which i can perform on the lustre client to conform lustre file system.

[Amitkumar Ghatwal]

Hi James A Simmons,

Are there any test cases which can be run to validate the lustre client configuration/performance when build lustre from source as done above ?

[Amitkumar Ghatwal]

Is there any inbuilt test-suite to validate lustre client ?

[James A Simmons]

Long fixed.