[LU-4435] Make lustre 2.4+ release rpm compatible with the ZFSonLinux repo

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.4.1, Lustre 2.4.2
• Type: Improvement
• Priority: Minor
• Reporter: Zhiqi Tao (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved

Environment

zfs-0.6.2-1.el6.x86_64, spl-0.6.2-1.el6.x86_64

Description

Symptom:

The instruction described in http://zfsonlinux.org/lustre.html works fine. I can create Lustre targets with ZFS backend and mount them.

However, when I tried to install lustre rpms from http://downloads.whamcloud.com/public/lustre/lustre-2.4.1 and http://downloads.whamcloud.com/public/lustre/lustre-2.4.2, I got the same error of Can't load module 'osd-zfs'.

– tried to mount the zfs lustre target –

[root@zfs-mds1 ~]# service lustre start mgt
Mounting lustre-mgt/mgt on /mnt/lustre/local/mgt
mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

Syslog shows

Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 158-c: Can't load module 'osd-zfs'
Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs
Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_config.c:374:class_attach()) Cannot create device MGS-osd of type osd-zfs : -19
Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:196:lustre_start_simple()) MGS-osd attach error -19
Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount_server.c:1682:server_fill_super()) Unable to start osd on lustre-mgt/mgt: -19
Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount  (-19)

– tried to load the osd-zfs module manually –

[root@zfs-oss1 ~]# insmod /lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko
insmod: error inserting '/lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko': -1 Unknown symbol in module

Syslog shows

Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_unregister
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_unregister
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_spill_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_spill_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_read
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_read
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_assign
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_assign
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_get
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_get
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_alloc
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_alloc
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_own
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_own
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_write
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_lookup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_return_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_return_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_assign_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_assign_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_object_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_object_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_callback_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_callback_register
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_setup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_setup
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_disown
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_disown
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa_create
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_ds
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_ds
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_spa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_spa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_get_dsl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_get_dsl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_userused_enabled
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_userused_enabled
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_zap
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_zap
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_get_txg
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_get_txg
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_request_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_request_arcbuf
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_commit
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_commit
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_callbacks
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_callbacks
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_hold_impl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_hold_impl
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_pool
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_pool
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_free_range
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_free_range
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_size
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_buf_hold
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_buf_hold
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_replace_all_by_template
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_replace_all_by_template
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_bonus
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_bonus
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_freeze
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_freeze
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_synced
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_synced
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_abort
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_abort
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_destroy
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_destroy
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_space
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_space
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_rele
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_set_blocksize
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_set_blocksize
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_update
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_free
Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_free

Impact:

This issue prevents me from trying the latest Lustre release but rather has to use the lustre and lustre-dkms rpms from the zfsonlinux repo. As of 01/04/2014, the lustre packages from the zfs repo is still 2.4

lustre-2.4.0-1.el6.x86_64
lustre-dkms-2.4.0-1.el6.noarch

Steps of reproducing the problem:

1. Install the patched Lustre kernel, kernel-firmware and kernel-devel from the Lustre 2.4.2 release

1. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-2.6.32-358.23.2.el6_lustre.x86_64.rpm
2. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-firmware-2.6.32-358.23.2.el6_lustre.x86_64.rpm
3. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-headers-2.6.32-358.23.2.el6_lustre.x86_64.rpm
4. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-devel-2.6.32-358.23.2.el6_lustre.x86_64.rpm

2. Reboot the Lustre server and verify that the Lustre patched kernel is in use.

1. uname -a
   Linux zfs-mds1 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

3. Install zfs packages from the zfsonlinux repo

1. yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
2. yum install zfs

4. Install Lustre module rpms

1. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
2. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-modules-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
3. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-osd-zfs-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm

5. Load the zfs, spl and lustre modules and verify no errors or warnings

1. modprobe -v spl
2. modprobe -v zfs
3. modprobe -v lustre

6. Create a Lustre MGT with ZFS as the back-end file system

1. mkfs.lustre --reformat --mgs --backfstype=zfs lustre-mgt/mgt mirror /dev/sdb /dev/sdc

  Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:  
Mount type: zfs
Flags:      0x64
             (MGS first_time update )
Persistent mount opts:
Parameters:

mkfs_cmd = zpool create -f -O canmount=off lustre-mgt mirror /dev/sdb /dev/sdc
mkfs_cmd = zfs create -o canmount=off -o xattr=sa lustre-mgt/mgt
Writing lustre-mgt/mgt properties
 lustre:version=1
 lustre:flags=100
 lustre:index=65535
 lustre:svname=MGS

7. mount the MGT target

1. zfs list
   NAME USED AVAIL REFER MOUNTPOINT
   lustre-mgt 1.08M 975M 30K /lustre-mgt
   lustre-mgt/mgt 979K 975M 979K /lustre-mgt/mgt

1. mkdir /mnt/mgt

1. mount -t lustre lustre-mgt/mgt /mnt/mgt

mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems

8. Check the syslog and see the error messages

Comments

[Bruno Faccini (Inactive)]

Hello Zhiqi, I think it is because the ZFS version you installed is ahead of the one we use to build osd-zfs (0.6.1?) and thus there are incompatibles/unknown symbols …

I think we could at least prevent/warn about it by adding more strict Require/dependency-check for osd-zfs RPM in lustre.spec.in. A follow-on to LU-2391 which only set a "zfs-kmod" rule without any version.

[Zhiqi Tao (Inactive)]

Hi Bruno,

Yes, indeed.

Thanks!

[Nathaniel Clark]

Lustre 2.4 is built with a slightly off master version of zfs. I have that specific branch exported to github here:

https://github.com/utopiabound/zfs/tree/lustre-zfs
https://github.com/utopiabound/spl/tree/lustre-zfs