[LU-3121] lustre-initialization-1: Can't load module 'osd-zfs'

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Affects Version/s: Lustre 2.4.1
• Labels: HB zfs
• Severity: 3
• Assignee: Chris Gearing (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Li Wei <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/418e2830-9f9f-11e2-9f27-52540035b04c.

The sub-test lustre-initialization_1 failed with the following error:

Test system failed to start single suite, so abandoning all hope and giving up

Info required for matching: lustre-initialization-1 lustre-initialization_1

For some unknown reason, the MDS couldn't load osd-zfs.ko:

08:17:45:LNet: Accept all, port 7988
08:17:45:LustreError: 158-c: Can't load module 'osd-zfs'
08:17:45:LustreError: 2944:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs
08:17:45:LustreError: 2944:0:(obd_config.c:374:class_attach()) Cannot create device lustre-MDT0000-osd of type osd-zfs : -19
08:17:45:LustreError: 2944:0:(obd_mount.c:196:lustre_start_simple()) lustre-MDT0000-osd attach error -19
08:17:45:LustreError: 2944:0:(obd_mount_server.c:1664:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -19
08:17:45:LustreError: 2944:0:(obd_mount.c:1264:lustre_fill_super()) Unable to mount  (-19)

Issue Links

• is related to: LU-2969 Fail to mount MDS as zfs on external OFED build

Comments

[Nathaniel Clark]

I believe this is fixed now. There is still some bug where lustre-inialization-1 fails, but it does not appear to be failing on loading osd-zfs.

[Nathaniel Clark]

zfs and ancillary packages are installed after kickstart, but lustre-osd-zfs isn't one of them.

[Nathaniel Clark]

+ yum install -y kernel-2.6.32-358.6.1.el6_lustre.x86_64 lustre-ldiskfs lustre-modules lustre lustre-tests
Loaded plugins: fastestmirror, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-358.6.1.el6_lustre will be installed
---> Package lustre.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed
--> Processing Dependency: lustre-osd for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64
--> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64
--> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64
--> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64
--> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64
---> Package lustre-ldiskfs.x86_64 0:4.1.0-2.6.32_358.6.1.el6_lustre.x86_64 will be installed
---> Package lustre-modules.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed
---> Package lustre-tests.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed
--> Running transaction check
---> Package lustre-osd-ldiskfs.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed
---> Package net-snmp-libs.x86_64 1:5.5-44.el6_4.1 will be installed
--> Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-44.el6_4.1.x86_64
--> Running transaction check
---> Package lm_sensors-libs.x86_64 0:3.1.1-17.el6 will be installed
--> Finished Dependency Resolution

The problem is that lustre requires lustre-osd and picks lustre-osd-ldiskfs thus not installing lustre-osd-zfs

The yum line should explicitly install lustre-osd-ldiskfs and lustre-osd-zfs, or we could add a virtual package lustre-all that requires both osd packages.

[Jian Yu]

Lustre Tag: v2_4_0_RC1
Lustre Build: http://build.whamcloud.com/job/lustre-master/1501/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs

Hit the issue again:
https://maloo.whamcloud.com/test_sets/7e89ecee-bf5a-11e2-88e0-52540035b04c

[Jian Yu]

Lustre Branch: master
Lustre Build: http://build.whamcloud.com/job/lustre-master/1495 (which contains the patch for LU-3269)
FSTYPE=zfs

The issue in this ticket still occurred:
https://maloo.whamcloud.com/test_sessions/754709ca-bd37-11e2-a548-52540035b04c

[Nathaniel Clark]

The above linked patch has landed for LU-3269.

[Nathaniel Clark]

If it's a module dependency ordering issue, it may be solved by http://review.whamcloud.com/6259 which fixes up lustre-osd-* / lustre-modules rpm dependencies.

[Nathaniel Clark]

I can reproduce these exact symptoms by installing all of lustre except lustre-osd-zfs normally, then install lustre-osd-zfs with --noscripts (causing it to skip running depmod). lustre-osd-zfs may skip depmod if the kernel isn't installed yet, still trying to figure out this exact vector.