[LU-8431] Cannot format zfs backend with lustre master build on CORAL zfs Created: 22/Jul/16 Updated: 14/Jun/18 Resolved: 24/Aug/16 |
|
| Status: | Closed |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | John Salinas (Inactive) | Assignee: | Nathaniel Clark |
| Resolution: | Not a Bug | Votes: | 0 |
| Labels: | LS_RZ | ||
| Severity: | 1 |
| Rank (Obsolete): | 9223372036854775807 |
| Description |
|
This bug is blocking testing and evaluation of the Lustre streaming feature for CORAL. mkfs.lustre --backfstype=zfs * results in: Failed to initialize ZFS library mkfs.lustre FATAL: unhandled/unloaded fs type 5 'zfs' mkfs.lustre FATAL: unable to prepare backend (22) mkfs.lustre: exiting with 22 (Invalid argument) Failed to initialize ZFS library mount.lustre: zfspool/mdt1 has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool Lustre build is form: ssh://hudson@review.whamcloud.com:29418/fs/lustre-release */master RPMs Running are: [root@wolf-3 mdisolation_step2_draid_testing]# rpm -qa |grep -i lustre lustre-osd-zfs-mount-2.8.55-1.el7.centos.x86_64 lustre-2.8.55-1.el7.centos.x86_64 lustre-tests-2.8.55-1.el7.centos.x86_64 kmod-lustre-2.8.55-1.el7.centos.x86_64 kmod-lustre-osd-zfs-2.8.55-1.el7.centos.x86_64 lustre-iokit-2.8.55-1.el7.centos.x86_64 kmod-lustre-tests-2.8.55-1.el7.centos.x86_64 [root@wolf-3 mdisolation_step2_draid_testing]# rpm -qa |grep -i zfs lustre-osd-zfs-mount-2.8.55-1.el7.centos.x86_64 libzfs2-0.6.5-1.el7.centos.x86_64 zfs-0.6.5-1.el7.centos.x86_64 kmod-lustre-osd-zfs-2.8.55-1.el7.centos.x86_64 zfs-test-0.6.5-1.el7.centos.x86_64 kmod-zfs-0.6.5-1.el7.centos.x86_64 The zfs is basically zfs master from last week + CORAL patches on top and can be found at fs/zfs -b coral-prototype |
| Comments |
| Comment by Nathaniel Clark [ 26/Jul/16 ] |
|
Do you have the lustre patches needed for building against ZFS master? http://review.whamcloud.com/19293 |
| Comment by John Salinas (Inactive) [ 26/Jul/16 ] |
|
That is all I am aware of right now – but until we can get testing running on this it it is hard to know if it is working as expected. Last night we figured out this is because the RPMs are not doing a modprobe zfs: sdc mirror /dev/sdd /dev/sdedex=0 --mdt --mgs zfspool/mdt1 mirror /dev/sdb /dev/s
execve("/sbin/mkfs.lustre", ["mkfs.lustre", "--backfstype=zfs", "--fsname=lsdraid", "--index=0", "--mdt", "--mgs", "zfspool/mdt1", "mirror", "/dev/sdb", "/dev/sdc", "mirror", "/dev/sdd", "/dev/sde"], [/* 28 vars */]) = 0
brk(0) = 0x1966000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0f12668000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=38295, ...}) = 0
mmap(NULL, 38295, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0f1265e000
close(3)
...etc...
access("/sys/module/zfs", F_OK) = -1 ENOENT (No such file or directory)
access("/sys/module/zfs", F_OK) = -1 ENOENT (No such file or directory)
write(2, "Failed to initialize ZFS library"..., 33Failed to initialize ZFS library
) = 33
munmap(0x7f0f0f8b5000, 2114232) = 0
munmap(0x7f0f0f670000, 2379656) = 0
munmap(0x7f0f0f46b000, 2114000) = 0
munmap(0x7f0f0ef5f000, 5289752) = 0
munmap(0x7f0f0ed4a000, 2180128) = 0
munmap(0x7f0f0eb37000, 2174328) = 0
munmap(0x7f0f0e6f2000, 2345800) = 0
munmap(0x7f0f12640000, 84225) = 0
munmap(0x7f0f0e4ed000, 2113920) = 0
munmap(0x7f0f0e2e8000, 2114112) = 0
munmap(0x7f0f0e0a1000, 2386184) = 0
munmap(0x7f0f0de8b000, 2184192) = 0
munmap(0x7f0f0dc86000, 2113904) = 0
munmap(0x7f0f0da70000, 2183520) = 0
munmap(0x7f0f0d860000, 2162024) = 0
write(2, "\nmkfs.lustre FATAL: ", 20
mkfs.lustre FATAL: ) = 20
write(2, "unhandled/unloaded fs type 5 'zf"..., 35unhandled/unloaded fs type 5 'zfs'
) = 35
write(2, "\nmkfs.lustre FATAL: ", 20
mkfs.lustre FATAL: ) = 20
write(2, "unable to prepare backend (22)\n", 31unable to prepare backend (22)
) = 31
write(2, "mkfs.lustre: exiting with 22 (In"..., 48mkfs.lustre: exiting with 22 (Invalid argument)
) = 48
exit_group(22) = ?
modprobe zfs
[ 992.594607] SPL: Loaded module v0.6.5-1
sdc mirror /dev/sdd /dev/sdedex=0 --mdt --mgs zfspool/mdt1 mirror /dev/sdb /dev/s
execve("/sbin/mkfs.lustre", ["mkfs.lustre", "--backfstype=zfs", "--fsname=lsdraid", "--index=0", "--mdt", "--mgs", "zfspool/mdt1", "mirror", "/dev/sdb", "/dev/sdc", "mirror", "/dev/sdd", "/dev/sde"], [/* 28 vars */]) = 0
...etc...
access("/sys/module/zfs", F_OK) = 0
access("/sys/module/zfs", F_OK) = 0
open("/dev/zfs", O_RDWR) = 3
close(3) = 0
open("/dev/zfs", O_RDWR) = 3
open("/etc/mtab", O_RDONLY) = 4
open("/etc/dfs/sharetab", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/dev/zfs", O_RDWR) = 5
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(4, 64), ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B115200 opost isig icanon echo ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c2072c000
write(1, "\n", 1
) = 1
write(1, " Permanent disk data:\n", 24 Permanent disk data:
) = 24
write(1, "Target: lsdraid:MDT0000\n", 28Target: lsdraid:MDT0000
) = 28
write(1, "Index: 0\n", 14Index: 0
) = 14
write(1, "Lustre FS: lsdraid\n", 20Lustre FS: lsdraid
) = 20
write(1, "Mount type: zfs\n", 16Mount type: zfs
) = 16
write(1, "Flags: 0x65\n", 17Flags: 0x65
) = 17
write(1, " (MDT MGS first_tim"..., 43 (MDT MGS first_time update )
) = 43
write(1, "Persistent mount opts: \n", 24Persistent mount opts:
) = 24
write(1, "Parameters:\n", 12Parameters:
) = 12
write(1, "\n", 1
) = 1
...etc..
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 24640
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f1c1f421670}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f1c1f421670}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24640, si_status=0, si_utime=0, si_stime=0} ---
unlink("/tmp/run_command_logueGkvh") = 0
ioctl(3, 0x5a12, 0x7fff54bff250) = 0
ioctl(3, 0x5a05, 0x7fff54bfbc00) = 0
brk(0) = 0xe59000
brk(0xe7a000) = 0xe7a000
write(1, "Writing zfspool/mdt1 properties\n", 32Writing zfspool/mdt1 properties
) = 32
write(1, " lustre:version=1\n", 19 lustre:version=1
) = 19
ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0
ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0
write(1, " lustre:flags=101\n", 19 lustre:flags=101
) = 19
ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0
ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0
write(1, " lustre:index=0\n", 17 lustre:index=0
) = 17
ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0
ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0
write(1, " lustre:fsname=lsdraid\n", 24 lustre:fsname=lsdraid
) = 24
ioctl(3, 0x5a16, 0x7fff54bfee20) = 0
ioctl(3, 0x5a12, 0x7fff54bfb800) = 0
write(1, " lustre:svname=lsdraid:MDT0000\n", 32 lustre:svname=lsdraid:MDT0000
) = 32
ioctl(3, 0x5a16, 0x7fff54bfee20) = 0
ioctl(3, 0x5a12, 0x7fff54bfb800) = 0
close(3) = 0
close(4) = 0
brk(0) = 0xe7a000
brk(0) = 0xe7a000
brk(0xe62000) = 0xe62000
brk(0) = 0xe62000
close(5) = 0
munmap(0x7f1c1d97a000, 2114232) = 0
munmap(0x7f1c1d735000, 2379656) = 0
munmap(0x7f1c1d530000, 2114000) = 0
munmap(0x7f1c1d024000, 5289752) = 0
munmap(0x7f1c1ce0f000, 2180128) = 0
munmap(0x7f1c1cbfc000, 2174328) = 0
munmap(0x7f1c1c7b7000, 2345800) = 0
munmap(0x7f1c20705000, 84225) = 0
munmap(0x7f1c1c5b2000, 2113920) = 0
munmap(0x7f1c1c3ad000, 2114112) = 0
munmap(0x7f1c1c166000, 2386184) = 0
munmap(0x7f1c1bf50000, 2184192) = 0
munmap(0x7f1c1bd4b000, 2113904) = 0
munmap(0x7f1c1bb35000, 2183520) = 0
munmap(0x7f1c1b925000, 2162024) = 0
exit_group(0)
|
| Comment by Nathaniel Clark [ 26/Jul/16 ] |
|
ZFS commit 87abfcba2283bfeb6636caf9aa2a72186d7708c4 for ZFS Issue #2556 changed the behavior of libzfs_load_module() that is called from libzfs_init() (called from zfs_init() in mkfs.lustre --backfstype=zfs). Either the environment variable ZFS_MODULE_LOADING=YES or we need to pre-load zfs. |
| Comment by Christopher Morrone [ 26/Jul/16 ] |
|
Yes, the proper approach now is to run "modprobe zfs" first. There were apparently long discussions, and it was felt that this is the most Linux-y approach. The environment variable may not remain in place long term, so it is probably best to just use modprobe. |
| Comment by John Salinas (Inactive) [ 26/Jul/16 ] |
|
Any reason why the rpm doesn't do the modprobe? |
| Comment by Christopher Morrone [ 26/Jul/16 ] |
|
That is not the sort of thing an rpm should do. rpm install scripts are a one-time operation. modprobe needs to be called potentially many times; any time the module is not loaded and we want to use it. |
| Comment by John Salinas (Inactive) [ 01/Aug/16 ] |
|
Could we use a similar method as Lustre or IB? It seems odd that at install and each boot I have to modprobe by hand as there is automatic method to do this. Or am I overlooking some admin setup that is part of ZOL already? Thanks! |
| Comment by Nathaniel Clark [ 24/Aug/16 ] |
|
John, Are you good with this issue? Do you need something else from this ticket? – |
| Comment by John Salinas (Inactive) [ 24/Aug/16 ] |
|
I am good – this ticket can be closed |