https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8431 Lustre <p>This bug is blocking testing and evaluation of the Lustre streaming feature for CORAL. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mkfs.lustre --backfstype=zfs * results in: Failed to initialize ZFS library mkfs.lustre FATAL: unhandled/unloaded fs type 5 'zfs' mkfs.lustre FATAL: unable to prepare backend (22) mkfs.lustre: exiting with 22 (Invalid argument) Failed to initialize ZFS library mount.lustre: zfspool/mdt1 has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool </pre> </div></div> <p>Lustre build is form: <tt>ssh://hudson@review.whamcloud.com:29418/fs/lustre-release */master</tt></p> <p>RPMs Running are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@wolf-3 mdisolation_step2_draid_testing]# rpm -qa |grep -i lustre lustre-osd-zfs-mount-2.8.55-1.el7.centos.x86_64 lustre-2.8.55-1.el7.centos.x86_64 lustre-tests-2.8.55-1.el7.centos.x86_64 kmod-lustre-2.8.55-1.el7.centos.x86_64 kmod-lustre-osd-zfs-2.8.55-1.el7.centos.x86_64 lustre-iokit-2.8.55-1.el7.centos.x86_64 kmod-lustre-tests-2.8.55-1.el7.centos.x86_64 [root@wolf-3 mdisolation_step2_draid_testing]# rpm -qa |grep -i zfs lustre-osd-zfs-mount-2.8.55-1.el7.centos.x86_64 libzfs2-0.6.5-1.el7.centos.x86_64 zfs-0.6.5-1.el7.centos.x86_64 kmod-lustre-osd-zfs-2.8.55-1.el7.centos.x86_64 zfs-test-0.6.5-1.el7.centos.x86_64 kmod-zfs-0.6.5-1.el7.centos.x86_64 </pre> </div></div> <p>The zfs is basically zfs master from last week + CORAL patches on top and can be found at fs/zfs -b coral-prototype </p> LU-8431

Cannot format zfs backend with lustre master build on CORAL zfs

Bug Major Closed Not a Bug Nathaniel Clark John Salinas LS_RZ Fri, 22 Jul 2016 21:40:48 +0000 Thu, 14 Jun 2018 21:39:48 +0000 Wed, 24 Aug 2016 21:55:32 +0000 0 6 <p>Do you have the lustre patches needed for building against ZFS master?</p> <p><a href="http://review.whamcloud.com/19293" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19293</a><br/> <a href="http://review.whamcloud.com/21251" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21251</a> - landed 7/20</p> <p>That is all I am aware of right now &#8211; but until we can get testing running on this it it is hard to know if it is working as expected. </p> <p>Last night we figured out this is because the RPMs are not doing a modprobe zfs: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sdc mirror /dev/sdd /dev/sdedex=0 --mdt --mgs zfspool/mdt1 mirror /dev/sdb /dev/s execve("/sbin/mkfs.lustre", ["mkfs.lustre", "--backfstype=zfs", "--fsname=lsdraid", "--index=0", "--mdt", "--mgs", "zfspool/mdt1", "mirror", "/dev/sdb", "/dev/sdc", "mirror", "/dev/sdd", "/dev/sde"], [/* 28 vars */]) = 0 brk(0) = 0x1966000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0f12668000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=38295, ...}) = 0 mmap(NULL, 38295, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0f1265e000 close(3) ...etc... access("/sys/module/zfs", F_OK) = -1 ENOENT (No such file or directory) access("/sys/module/zfs", F_OK) = -1 ENOENT (No such file or directory) write(2, "Failed to initialize ZFS library"..., 33Failed to initialize ZFS library ) = 33 munmap(0x7f0f0f8b5000, 2114232) = 0 munmap(0x7f0f0f670000, 2379656) = 0 munmap(0x7f0f0f46b000, 2114000) = 0 munmap(0x7f0f0ef5f000, 5289752) = 0 munmap(0x7f0f0ed4a000, 2180128) = 0 munmap(0x7f0f0eb37000, 2174328) = 0 munmap(0x7f0f0e6f2000, 2345800) = 0 munmap(0x7f0f12640000, 84225) = 0 munmap(0x7f0f0e4ed000, 2113920) = 0 munmap(0x7f0f0e2e8000, 2114112) = 0 munmap(0x7f0f0e0a1000, 2386184) = 0 munmap(0x7f0f0de8b000, 2184192) = 0 munmap(0x7f0f0dc86000, 2113904) = 0 munmap(0x7f0f0da70000, 2183520) = 0 munmap(0x7f0f0d860000, 2162024) = 0 write(2, "\nmkfs.lustre FATAL: ", 20 mkfs.lustre FATAL: ) = 20 write(2, "unhandled/unloaded fs type 5 'zf"..., 35unhandled/unloaded fs type 5 'zfs' ) = 35 write(2, "\nmkfs.lustre FATAL: ", 20 mkfs.lustre FATAL: ) = 20 write(2, "unable to prepare backend (22)\n", 31unable to prepare backend (22) ) = 31 write(2, "mkfs.lustre: exiting with 22 (In"..., 48mkfs.lustre: exiting with 22 (Invalid argument) ) = 48 exit_group(22) = ? modprobe zfs [ 992.594607] SPL: Loaded module v0.6.5-1 sdc mirror /dev/sdd /dev/sdedex=0 --mdt --mgs zfspool/mdt1 mirror /dev/sdb /dev/s execve("/sbin/mkfs.lustre", ["mkfs.lustre", "--backfstype=zfs", "--fsname=lsdraid", "--index=0", "--mdt", "--mgs", "zfspool/mdt1", "mirror", "/dev/sdb", "/dev/sdc", "mirror", "/dev/sdd", "/dev/sde"], [/* 28 vars */]) = 0 ...etc... access("/sys/module/zfs", F_OK) = 0 access("/sys/module/zfs", F_OK) = 0 open("/dev/zfs", O_RDWR) = 3 close(3) = 0 open("/dev/zfs", O_RDWR) = 3 open("/etc/mtab", O_RDONLY) = 4 open("/etc/dfs/sharetab", O_RDONLY) = -1 ENOENT (No such file or directory) open("/dev/zfs", O_RDWR) = 5 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(4, 64), ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B115200 opost isig icanon echo ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c2072c000 write(1, "\n", 1 ) = 1 write(1, " Permanent disk data:\n", 24 Permanent disk data: ) = 24 write(1, "Target: lsdraid:MDT0000\n", 28Target: lsdraid:MDT0000 ) = 28 write(1, "Index: 0\n", 14Index: 0 ) = 14 write(1, "Lustre FS: lsdraid\n", 20Lustre FS: lsdraid ) = 20 write(1, "Mount type: zfs\n", 16Mount type: zfs ) = 16 write(1, "Flags: 0x65\n", 17Flags: 0x65 ) = 17 write(1, " (MDT MGS first_tim"..., 43 (MDT MGS first_time update ) ) = 43 write(1, "Persistent mount opts: \n", 24Persistent mount opts: ) = 24 write(1, "Parameters:\n", 12Parameters: ) = 12 write(1, "\n", 1 ) = 1 ...etc.. &lt;... wait4 resumed&gt; [{WIFEXITED(s) &amp;&amp; WEXITSTATUS(s) == 0}], 0, NULL) = 24640 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f1c1f421670}, NULL, 8) = 0 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f1c1f421670}, NULL, 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24640, si_status=0, si_utime=0, si_stime=0} --- unlink("/tmp/run_command_logueGkvh") = 0 ioctl(3, 0x5a12, 0x7fff54bff250) = 0 ioctl(3, 0x5a05, 0x7fff54bfbc00) = 0 brk(0) = 0xe59000 brk(0xe7a000) = 0xe7a000 write(1, "Writing zfspool/mdt1 properties\n", 32Writing zfspool/mdt1 properties ) = 32 write(1, " lustre:version=1\n", 19 lustre:version=1 ) = 19 ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0 ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0 write(1, " lustre:flags=101\n", 19 lustre:flags=101 ) = 19 ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0 ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0 write(1, " lustre:index=0\n", 17 lustre:index=0 ) = 17 ioctl(3, 0x5a16, 0x7fff54bfedc0) = 0 ioctl(3, 0x5a12, 0x7fff54bfb7a0) = 0 write(1, " lustre:fsname=lsdraid\n", 24 lustre:fsname=lsdraid ) = 24 ioctl(3, 0x5a16, 0x7fff54bfee20) = 0 ioctl(3, 0x5a12, 0x7fff54bfb800) = 0 write(1, " lustre:svname=lsdraid:MDT0000\n", 32 lustre:svname=lsdraid:MDT0000 ) = 32 ioctl(3, 0x5a16, 0x7fff54bfee20) = 0 ioctl(3, 0x5a12, 0x7fff54bfb800) = 0 close(3) = 0 close(4) = 0 brk(0) = 0xe7a000 brk(0) = 0xe7a000 brk(0xe62000) = 0xe62000 brk(0) = 0xe62000 close(5) = 0 munmap(0x7f1c1d97a000, 2114232) = 0 munmap(0x7f1c1d735000, 2379656) = 0 munmap(0x7f1c1d530000, 2114000) = 0 munmap(0x7f1c1d024000, 5289752) = 0 munmap(0x7f1c1ce0f000, 2180128) = 0 munmap(0x7f1c1cbfc000, 2174328) = 0 munmap(0x7f1c1c7b7000, 2345800) = 0 munmap(0x7f1c20705000, 84225) = 0 munmap(0x7f1c1c5b2000, 2113920) = 0 munmap(0x7f1c1c3ad000, 2114112) = 0 munmap(0x7f1c1c166000, 2386184) = 0 munmap(0x7f1c1bf50000, 2184192) = 0 munmap(0x7f1c1bd4b000, 2113904) = 0 munmap(0x7f1c1bb35000, 2183520) = 0 munmap(0x7f1c1b925000, 2162024) = 0 exit_group(0) </pre> </div></div> <p>ZFS commit 87abfcba2283bfeb6636caf9aa2a72186d7708c4 for <a href="https://github.com/zfsonlinux/zfs/issues/2556" class="external-link" target="_blank" rel="nofollow noopener">ZFS Issue #2556</a> changed the behavior of libzfs_load_module() that is called from libzfs_init() (called from zfs_init() in <tt>mkfs.lustre --backfstype=zfs</tt>). Either the environment variable <tt>ZFS_MODULE_LOADING=YES</tt> or we need to pre-load zfs.</p> <p>Yes, the proper approach now is to run "modprobe zfs" first. There were apparently long discussions, and it was felt that this is the most Linux-y approach. The environment variable may not remain in place long term, so it is probably best to just use modprobe.</p> <p>Any reason why the rpm doesn't do the modprobe? </p> <p>That is not the sort of thing an rpm should do. rpm install scripts are a one-time operation. modprobe needs to be called potentially many times; any time the module is not loaded and we want to use it.</p> <p>Could we use a similar method as Lustre or IB? It seems odd that at install and each boot I have to modprobe by hand as there is automatic method to do this. Or am I overlooking some admin setup that is part of ZOL already? Thanks!</p> <p>John,</p> <p>Are you good with this issue? Do you need something else from this ticket?</p> <p>&#8211;<br/> Nathaniel</p> <p>I am good &#8211; this ticket can be closed</p> Development Rank 1|hzyic7: Rank (Obsolete) 9223372036854775807 Severity