Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4435

Make lustre 2.4+ release rpm compatible with the ZFSonLinux repo

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • Lustre 2.4.1, Lustre 2.4.2
    • None
    • zfs-0.6.2-1.el6.x86_64, spl-0.6.2-1.el6.x86_64
    • 12179

    Description

      Symptom:

      The instruction described in http://zfsonlinux.org/lustre.html works fine. I can create Lustre targets with ZFS backend and mount them.

      However, when I tried to install lustre rpms from http://downloads.whamcloud.com/public/lustre/lustre-2.4.1 and http://downloads.whamcloud.com/public/lustre/lustre-2.4.2, I got the same error of Can't load module 'osd-zfs'.

      – tried to mount the zfs lustre target –

      [root@zfs-mds1 ~]# service lustre start mgt
      Mounting lustre-mgt/mgt on /mnt/lustre/local/mgt
      mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
      Are the lustre modules loaded?
      Check /etc/modprobe.conf and /proc/filesystems

      Syslog shows

      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 158-c: Can't load module 'osd-zfs'
      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs
      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_config.c:374:class_attach()) Cannot create device MGS-osd of type osd-zfs : -19
      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:196:lustre_start_simple()) MGS-osd attach error -19
      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount_server.c:1682:server_fill_super()) Unable to start osd on lustre-mgt/mgt: -19
      Jan  4 21:22:10 zfs-mds1 kernel: LustreError: 12047:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount  (-19)

      – tried to load the osd-zfs module manually –

      [root@zfs-oss1 ~]# insmod /lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko
      insmod: error inserting '/lib/modules/2.6.32-358.23.2.el6_lustre.x86_64/updates/kernel/fs/lustre/osd_zfs.ko': -1 Unknown symbol in module

      Syslog shows

      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_register
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_register
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_write
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_write
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dsl_prop_unregister
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dsl_prop_unregister
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_spill_rele
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_spill_rele
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_read
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_read
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_assign
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_assign
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_get
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_get
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_create
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_create
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_lookup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_lookup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_alloc
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_alloc
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_free
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_free
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_own
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_own
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_write
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_write
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_lookup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_lookup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_return_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_return_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_assign_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_assign_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_object_size
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_object_size
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_callback_register
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_callback_register
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_setup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_setup
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_update
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_update
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_disown
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_disown
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa_create
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa_create
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_ds
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_ds
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_spa
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_spa
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_get_dsl
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_get_dsl
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_userused_enabled
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_userused_enabled
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_zap
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_zap
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_get_txg
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_get_txg
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_request_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_request_arcbuf
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_commit
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_commit
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_callbacks
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_callbacks
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_sa
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_sa
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_hold_impl
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_hold_impl
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_pool
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_pool
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_free_range
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_free_range
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_size
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_size
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_buf_hold
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_buf_hold
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_replace_all_by_template
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_replace_all_by_template
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_bonus
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_bonus
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol spa_freeze
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol spa_freeze
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol txg_wait_synced
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol txg_wait_synced
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_abort
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_abort
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_handle_destroy
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_handle_destroy
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_objset_space
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_objset_space
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dbuf_rele
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dbuf_rele
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_object_set_blocksize
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_object_set_blocksize
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol sa_bulk_update
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol sa_bulk_update
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: disagrees about version of symbol dmu_tx_hold_free
      Jan 4 22:15:33 zfs-mds1 kernel: osd_zfs: Unknown symbol dmu_tx_hold_free

      Impact:

      This issue prevents me from trying the latest Lustre release but rather has to use the lustre and lustre-dkms rpms from the zfsonlinux repo. As of 01/04/2014, the lustre packages from the zfs repo is still 2.4

      lustre-2.4.0-1.el6.x86_64
      lustre-dkms-2.4.0-1.el6.noarch

      Steps of reproducing the problem:

      1. Install the patched Lustre kernel, kernel-firmware and kernel-devel from the Lustre 2.4.2 release

      1. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-2.6.32-358.23.2.el6_lustre.x86_64.rpm
      2. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-firmware-2.6.32-358.23.2.el6_lustre.x86_64.rpm
      3. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-headers-2.6.32-358.23.2.el6_lustre.x86_64.rpm
      4. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/kernel-devel-2.6.32-358.23.2.el6_lustre.x86_64.rpm

      2. Reboot the Lustre server and verify that the Lustre patched kernel is in use.

      1. uname -a
        Linux zfs-mds1 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

      3. Install zfs packages from the zfsonlinux repo

      1. yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
      2. yum install zfs

      4. Install Lustre module rpms

      1. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
      2. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-modules-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm
      3. yum localinstall --nogpgcheck http://downloads.whamcloud.com/public/lustre/lustre-2.4.2/el6/server/RPMS/x86_64/lustre-osd-zfs-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64.rpm

      5. Load the zfs, spl and lustre modules and verify no errors or warnings

      1. modprobe -v spl
      2. modprobe -v zfs
      3. modprobe -v lustre

      6. Create a Lustre MGT with ZFS as the back-end file system

      1. mkfs.lustre --reformat --mgs --backfstype=zfs lustre-mgt/mgt mirror /dev/sdb /dev/sdc

        Permanent disk data:
      Target:     MGS
      Index:      unassigned
      Lustre FS:  
      Mount type: zfs
      Flags:      0x64
                   (MGS first_time update )
      Persistent mount opts:
      Parameters:

      mkfs_cmd = zpool create -f -O canmount=off lustre-mgt mirror /dev/sdb /dev/sdc
      mkfs_cmd = zfs create -o canmount=off -o xattr=sa lustre-mgt/mgt
      Writing lustre-mgt/mgt properties
       lustre:version=1
       lustre:flags=100
       lustre:index=65535
       lustre:svname=MGS

      7. mount the MGT target

      1. zfs list
        NAME USED AVAIL REFER MOUNTPOINT
        lustre-mgt 1.08M 975M 30K /lustre-mgt
        lustre-mgt/mgt 979K 975M 979K /lustre-mgt/mgt
      1. mkdir /mnt/mgt
      1. mount -t lustre lustre-mgt/mgt /mnt/mgt

      mount.lustre: mount lustre-mgt/mgt at /mnt/lustre/local/mgt failed: No such device
      Are the lustre modules loaded?
      Check /etc/modprobe.conf and /proc/filesystems

      8. Check the syslog and see the error messages

      Attachments

        Activity

          People

            wc-triage WC Triage
            zhiqi Zhiqi Tao (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: