[LU-4435] Make lustre 2.4+ release rpm compatible with the ZFSonLinux repo

Thu, 23 Jan 2014 20:03:39 +0000

Symptom:

The instruction described in http://zfsonlinux.org/lustre.html works fine. I can create Lustre targets with ZFS backend and mount them.

However, when I tried to install lustre rpms from http://downloads.whamcloud.com/public/lustre/lustre-2.4.1 and http://downloads.whamcloud.com/public/lustre/lustre-2.4.2, I got the same error of Can't load module 'osd-zfs'.

– tried to mount the zfs lustre target –

[root@zfs-mds1 ~]# service lustre start mgt
Mounting lustre-mgt/mgt on /mnt/lustre/local/mgt
mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

Syslog shows

Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 158-c: Can't load module 'osd-zfs'
Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs
Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_config.c:374:class_attach()) Cannot create device MGS-osd of type osd-zfs : -19
Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:196:lustre_start_simple()) MGS-osd attach error -19
Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount_server.c:1682:server_fill_super()) Unable to start osd on lustre-mgt/mgt: -19
Jan 4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount (-19)

– tried to load the osd-zfs module manually –

[root@zfs-oss1 ~]# insmod /lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko
insmod: error inserting '/lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko': -1 Unknown symbol in module

Syslog shows

Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_unregister
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_unregister
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_spill_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_spill_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_read
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_read
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_assign
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_assign
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_get
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_get
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_alloc
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_alloc
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_own
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_own
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_return_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_return_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_assign_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_assign_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_object_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_object_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_callback_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_callback_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_setup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_setup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_disown
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_disown
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_ds
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_ds
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_spa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_spa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_get_dsl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_get_dsl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_userused_enabled
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_userused_enabled
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_zap
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_zap
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_get_txg
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_get_txg
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_request_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_request_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_commit
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_commit
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_callbacks
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_callbacks
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_hold_impl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_hold_impl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_pool
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_pool
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_free_range
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_free_range
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_buf_hold
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_buf_hold
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_replace_all_by_template
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_replace_all_by_template
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_bonus
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_bonus
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_freeze
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_freeze
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_synced
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_synced
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_abort
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_abort
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_destroy
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_destroy
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_space
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_space
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_set_blocksize
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_set_blocksize
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_free

Impact:

This issue prevents me from trying the latest Lustre release but rather has to use the lustre and lustre-dkms rpms from the zfsonlinux repo. As of 01/04/2014, the lustre packages from the zfs repo is still 2.4

lustre-2.4.0-1.el6.x86_64
lustre-dkms-2.4.0-1.el6.noarch

Steps of reproducing the problem:

1. Install the patched Lustre kernel, kernel-firmware and kernel-devel from the Lustre 2.4.2 release

yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-2.6.32-358.23.2.el6_lustre.x86_64.rpm
yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-firmware-2.6.32-358.23.2.el6_lustre.x86_64.rpm
yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-headers-2.6.32-358.23.2.el6_lustre.x86_64.rpm
yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-devel-2.6.32-358.23.2.el6_lustre.x86_64.rpm

2. Reboot the Lustre server and verify that the Lustre patched kernel is in use.

uname -a
Linux zfs-mds1 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

3. Install zfs packages from the zfsonlinux repo

yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
yum install zfs

4. Install Lustre module rpms

yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-modules-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-osd-zfs-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm

5. Load the zfs, spl and lustre modules and verify no errors or warnings

modprobe -v spl
modprobe -v zfs
modprobe -v lustre

6. Create a Lustre MGT with ZFS as the back-end file system

mkfs.lustre --reformat --mgs --backfstype=zfs lustre-mgt/mgt mirror /dev/sdb /dev/sdc

  Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:
Mount type: zfs
Flags:      0x64
             (MGS first_time update )
Persistent mount opts:
Parameters:

mkfs_cmd = zpool create -f -O canmount=off lustre-mgt mirror /dev/sdb /dev/sdc
mkfs_cmd = zfs create -o canmount=off -o xattr=sa lustre-mgt/mgt
Writing lustre-mgt/mgt properties
lustre:version=1
lustre:flags=100
lustre:index=65535
lustre:svname=MGS

7. mount the MGT target

zfs list
NAME USED AVAIL REFER MOUNTPOINT
lustre-mgt 1.08M 975M 30K /lustre-mgt
lustre-mgt/mgt 979K 975M 979K /lustre-mgt/mgt

mkdir /mnt/mgt

mount -t lustre lustre-mgt/mgt /mnt/mgt

mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

8. Check the syslog and see the error messages

Whamcloud Community JIRA

[LU-4435] Make lustre 2.4+ release rpm compatible with the ZFSonLinux repo