[LU-4491] Obdfilter survey failed on Lustre ZFS OST

Thu, 16 Jan 2014 05:29:49 +0000

Symptom:

Obdfilter-survey test failed to run after completing one test.

[root@regal-oss01 obdfilter-survey]# ./run_obdfilter-survey.sh
Wed Jan 15 10:00:00 CST 2014 Obdfilter-survey for case=disk from regal-oss01
ost 1 sz 16777216K rsz 1024K obj 1 thr 1 write 182.16 [ 3.00, 296.96] rewrite 25.02 [ 8.00, 31.00] read 345.64 [ 263.96,1069.88]
ost 1 sz 16777216K rsz 1024K obj 1 thr 2 ERROR: 1 != 0
create: 1 objects
error: create: #1 - File exists
created object #s on regal-oss01:lustre-OST0000_ecc not contiguous

Jan 15 09:59:10 regal-oss01 kernel: Lustre: lustre-OST0000: new disk, initializing
Jan 15 09:59:10 regal-oss01 kernel: Lustre: srv-lustre-OST0000: No data found on store. Initialize space
Jan 15 10:00:00 regal-oss01 kernel: Lustre: Echo OBD driver; http://www.lustre.org/
Jan 15 10:01:01 regal-oss01 kernel: ERST: NVRAM ERST Log Address Range is not implemented yet
Jan 15 10:13:26 regal-oss01 kernel: LustreError: 4973:0:(osd_handler.c:213:osd_trans_start()) lustre-OST0000: can't assign tx: rc = -17
Jan 15 10:13:26 regal-oss01 kernel: LustreError: 4973:0:(ofd_obd.c:1356:ofd_create()) lustre-OST0000: unable to precreate: rc = -17
Jan 15 10:13:26 regal-oss01 kernel: LustreError: 4973:0:(echo_client.c:2305:echo_create_object()) Cannot create objects: rc = -17
Jan 15 10:13:26 regal-oss01 kernel: LustreError: 4973:0:(echo_client.c:2329:echo_create_object()) create object failed with: rc = -17

Steps to reproduce:

1. Install the Lustre and ZFS rpms from http://build.whamcloud.com/job/lustre-b2_5/2/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/

[root@regal-oss01 lustre_2.5_ga]# ls
dkms-2.2.0.3-1.noarch.rpm
e2fsprogs-1.42.7.wc2-7.el6.x86_64.rpm
e2fsprogs-libs-1.42.7.wc2-7.el6.x86_64.rpm
e2fsprogs-static-1.42.7.wc2-7.el6.x86_64.rpm
kernel-2.6.32-358.18.1.el6_lustre.x86_64.rpm
kernel-devel-2.6.32-358.18.1.el6_lustre.x86_64.rpm
kernel-firmware-2.6.32-358.18.1.el6_lustre.x86_64.rpm
kernel-headers-2.6.32-358.18.1.el6_lustre.x86_64.rpm
kmod-spl-2.6.32-358.18.1.el6_lustre.x86_64-0.6.2-1.el6.x86_64.rpm
kmod-spl-devel-0.6.2-1.el6.x86_64.rpm
kmod-spl-devel-2.6.32-358.18.1.el6_lustre.x86_64-0.6.2-1.el6.x86_64.rpm
kmod-zfs-2.6.32-358.18.1.el6_lustre.x86_64-0.6.2-1.el6.x86_64.rpm
kmod-zfs-devel-0.6.2-1.el6.x86_64.rpm
kmod-zfs-devel-2.6.32-358.18.1.el6_lustre.x86_64-0.6.2-1.el6.x86_64.rpm
libcom_err-1.42.7.wc2-7.el6.x86_64.rpm
libss-1.42.7.wc2-7.el6.x86_64.rpm
lustre-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-iokit-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-modules-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-osd-ldiskfs-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-osd-zfs-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-source-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
lustre-tests-2.5.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64.rpm
perf-2.6.32-358.18.1.el6_lustre.x86_64.rpm
python-perf-2.6.32-358.18.1.el6_lustre.x86_64.rpm
spl-0.6.2-1.el6.x86_64.rpm
spl-dkms-0.6.2-1.el6.noarch.rpm
zfs-0.6.2-1.el6.x86_64.rpm
zfs-devel-0.6.2-1.el6.x86_64.rpm
zfs-dkms-0.6.2-1.el6.noarch.rpm
zfs-dracut-0.6.2-1.el6.x86_64.rpm
zfs-test-0.6.2-1.el6.x86_64.rpm

2. Configure MGS and MDT on one node

3. Configure one zfs OST with 12 disks on the other node.

[root@regal-oss01 ~]# mkfs.lustre --ost --mgsnode=10.10.187.20@o2ib --fsname=lustre --index=0 --backfstype=zfs --reformat ost/ost0 raidz2 mpathc mpathd mpathe mpathf mpathg mpathh mpathi mpathj mpathk mpathl spare mpathm mpathn

Permanent disk data:
Target: lustre:OST0000
Index: 0
Lustre FS: lustre
Mount type: zfs
Flags: 0x62
(OST first_time update )
Persistent mount opts:
Parameters: mgsnode=10.10.187.20@o2ib

mkfs_cmd = zfs create -o canmount=off -o xattr=sa ost/ost0
Writing ost/ost0 properties
lustre:version=1
lustre:flags=98
lustre:index=0
lustre:fsname=lustre
lustre:svname=lustre:OST0000
lustre:mgsnode=10.10.187.20@o2ib
[root@regal-oss01 ~]# mount -t lustre ost/ost0 /mnt/ost0
[root@regal-oss01 ~]# lctl dl
0 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5
1 UP mgc MGC10.10.187.20@o2ib d09cdeb5-680c-d157-2c26-dc5e43ccb6ea 5
2 UP ost OSS OSS_uuid 3
3 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 3
4 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5

4. Run obdfilter-survey test.

thrhi=1024 thrlo=1 nobjhi=1024 nobjlo=1 size=16384 case=disk targets=regal-oss01:lustre-OST0000 rslt_loc=/root/obdfilter-survey/ obdfilter-survey

Whamcloud Community JIRA

[LU-4491] Obdfilter survey failed on Lustre ZFS OST