[LU-11071] Support for Lustre Servers on Ubuntu 18.04 LTS

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.12.0
• Type: New Feature
• Priority: Minor
• Reporter: Li Dongyang (Inactive)
• Assignee: Dongyang Li
• Resolution: Fixed

Description

Ubuntu 18.04 LTS ships kernel 4.15, with the client support for 4.15 landed to master,

Now we can add 4.15 server support.

Note that the new e2fsprogs rebasing is still in progress, the testing has done with the current e2fsprogs 1.42.13.wc6

Issue Links

• is related to: LU-11044 ext4_dir_operations uses iterate_shar...
• is related to: LU-11360 WARNING at /build/linux-5s7Xkn/linux-...
• is related to: LU-9564 Support for Lustre Servers on Ubuntu ...
• is related to: LU-10942 Support for Lustre Servers (ldiskfs b...
• is related to: LU-11368 utils linking issue on ubuntu18.04
• is related to: LU-10997 Ubuntu 18 support
• is related to: LU-11440 Make e2fsprogs-1.44.3-wc1 release

Comments

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32613
Subject: LU-11071 build: Add server build support for Ubuntu 18.04
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f9aa720b3de52f50eb0b4a4dd80fe59c543aa41e

[Bob Glossman (Inactive)]

this patch isn't working for me at all. when I apply it & try to build I get errors in osd-ldiskfs:

  CC [M]  /home/bogl/lustre-release/lustre/osd-ldiskfs/osd_iam_lfix.o
  CC [M]  /home/bogl/lustre-release/lustre/osd-ldiskfs/osd_iam_lvar.o
  CC [M]  /home/bogl/lustre-release/lustre/osd-ldiskfs/osd_io.o
/home/bogl/lustre-release/lustre/osd-ldiskfs/osd_io.c: In function ‘osd_do_bio’:
/home/bogl/lustre-release/lustre/osd-ldiskfs/osd_io.c:380:9: error: implicit declaration of function ‘bio_integrity_enabled’; did you mean ‘bio_integrity_flagged’? [-Werror=implicit-function-declaration]
     if (bio_integrity_enabled(bio)) {
         ^~~~~~~~~~~~~~~~~~~~~
         bio_integrity_flagged
cc1: all warnings being treated as errors
scripts/Makefile.build:332: recipe for target '/home/bogl/lustre-release/lustre/osd-ldiskfs/osd_io.o' failed
make[6]: *** [/home/bogl/lustre-release/lustre/osd-ldiskfs/osd_io.o] Error 1
scripts/Makefile.build:606: recipe for target '/home/bogl/lustre-release/lustre/osd-ldiskfs' failed
make[5]: *** [/home/bogl/lustre-release/lustre/osd-ldiskfs] Error 2
scripts/Makefile.build:606: recipe for target '/home/bogl/lustre-release/lustre' failed
make[4]: *** [/home/bogl/lustre-release/lustre] Error 2
Makefile:1552: recipe for target '_module_/home/bogl/lustre-release' failed
make[3]: *** [_module_/home/bogl/lustre-release] Error 2
make[3]: Leaving directory '/usr/src/linux-headers-4.15.0-22-generic'
autoMakefile:1054: recipe for target 'modules' failed
make[2]: *** [modules] Error 2
make[2]: Leaving directory '/home/bogl/lustre-release'
autoMakefile:595: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/bogl/lustre-release'
autoMakefile:483: recipe for target 'all' failed
make: *** [all] Error 2

[Bob Glossman (Inactive)]

Calls to bio_integrity_enabled were added only recently by the landing of the mod for "LU-10472 osd-ldiskfs: add T10PI support for BIO"

[Li Dongyang (Inactive)]

we need https://review.whamcloud.com/#/c/32621/ to address the T10PI BIO issue

[Bob Glossman (Inactive)]

I confirm that applying https://review.whamcloud.com/32621 fixes the basic build problem. 'make' now works. However 'make debs' still fails. errors:

dpkg-deb: building package 'lustre-source' in '../lustre-source_2.11.52-42-g0460ff4-1_all.deb'.
dh_testdir
dh_testroot
dh_installdirs -p lustre-server-utils
dh_installdocs -p  lustre-server-utils
dh_installman -p lustre-server-utils
dh_install -p lustre-server-utils
dh_install: Cannot find (any matches for) "debian/tmp/lib/systemd/system/*" (tried in ., debian/tmp)

dh_install: lustre-server-utils missing files: debian/tmp/lib/systemd/system/*
dh_install: missing files, aborting
debian/rules:201: recipe for target 'binary-lustre-server-utils' failed
make[1]: *** [binary-lustre-server-utils] Error 25
make[1]: Leaving directory '/home/bogl/lustre-release'
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
autoMakefile:1191: recipe for target 'debs' failed
make: *** [debs] Error 2

[Li Dongyang (Inactive)]

hm, I can build the deb packages without problems, do you have dh-systemd installed?

Another issue with the debian builds you might encounter is if you only enable ldiskfs backends, it expects /etc/ldev.conf which is only installed with zfs is enabled.

You can walk around the problem either by getting rid of it in lustre-server-utils.conffiles, or enable osd-zfs. I'm still figuring out how to handle this properly.

[James A Simmons]

I can give it a look. Did you update the patch with Yang feedback?

[Li Dongyang (Inactive)]

Thanks James, I'm working on the new patch set.

[Bob Glossman (Inactive)]

dh-systemd looks relatively content free. dpkg -L shows it has only some doc files. I tried installing it, no difference. build still fails the same.

[Li Dongyang (Inactive)]

Bob, I think I found out what the problem is:

checking for pkg-config... no
checking systemd unit file directory... ./configure: line 33500: --variable=systemdsystemunitdir: command not found
no

please install pkgconf and run configure again

[James A Simmons]

Do we need to make pkg-config a package requirement?

[Li Dongyang (Inactive)]

I think we should, James if you have a patch to fix the conffiles issue, please add this as well, thanks

[James A Simmons]

I'm swamped until next week Tuesday. I will look at it then.

[Bob Glossman (Inactive)]

please install pkgconf and run configure again

I installed both pkg-config and pkgconf. this makes the error you pointed at go away. But the build still fails. now the errors look like:

dh_fixperms -p lustre-server-utils
dh_gencontrol -p lustre-server-utils
dpkg-gencontrol: warning: Depends field of package lustre-server-utils: unknown substitution variable ${shlibs:Depends}
dpkg-gencontrol: warning: Depends field of package lustre-server-utils: unknown substitution variable ${shlibs:Depends}
dh_md5sums -p lustre-server-utils
dh_builddeb -p lustre-server-utils
dpkg-deb: error: conffile '/etc/ldev.conf' does not appear in package
dh_builddeb: dpkg-deb --build debian/lustre-server-utils .. returned exit code 2
dpkg-deb: building package 'lustre-server-utils-dbgsym' in 'debian/.debhelper/scratch-space/build-lustre-server-utils/lustre-server-utils-dbgsym_2.11.52-42-g0460ff4-dirty-1_amd64.deb'.
	Renaming lustre-server-utils-dbgsym_2.11.52-42-g0460ff4-dirty-1_amd64.deb to lustre-server-utils-dbgsym_2.11.52-42-g0460ff4-dirty-1_amd64.ddeb
dh_builddeb: Aborting due to earlier error
debian/rules:201: recipe for target 'binary-lustre-server-utils' failed
make[1]: *** [binary-lustre-server-utils] Error 255
make[1]: Leaving directory '/home/bogl/lustre-release'
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
autoMakefile:1194: recipe for target 'debs' failed
make: *** [debs] Error 2

Strongly suspect this is the build failure you mentioned in your earlier comment:

Another issue with the debian builds you might encounter is if you only enable ldiskfs backends, it expects /etc/ldev.conf which is only installed with zfs is enabled.

[James A Simmons]

Okay I working on a solution for the ldev.conf being included even when zfs is disabled.

[James A Simmons]

I have made a lot of progress.

1) The conffiles is actually easy to resolve. Just remove them. Newer debhelper will automatically create      conffiles when it sees any files in the /etc tree. 

2) Ubuntu has a /lib/modules/$(uname)/build symlink but not a source symlink. This meant that $LINUX which is meant to point to a linux source tree ended up pointing to linux-header-$(uname)generic instead. This broke the build. The solution was to add test for linux-source$(uname) before the module symlink.

3) Using the real source tree exposed the breakage with setting IB_OPTIONS. Its was set to either "no" or the IB source path, which is the linux source tree. Well Ubuntu stores Module.symvers in the linux-obj area instead. This caused various breakage as well. Now the default behavior is to handle the --with-o2ib="yes" case.

4) Missing resource-agent support.

5) What still needs to be fixed is ZFS. This is NOT totally fixed. Currently you see this output:

checking spl source directory... /usr/src/linux-source-4.15.0/spl

checking spl build directory... ls: cannot access '/usr/src/linux-source-4.15.0/spl/[0-9]*/': No such file or directory

Not found

checking zfs source directory... /usr/src/linux-source-4.15.0/zfs

checking zfs build directory... ls: cannot access '/usr/src/linux-source-4.15.0/zfs/[0-9]*/': No such file or directory

Not found

This is due to the check for [spl|zfs]_config.h which does not exist on Ubuntu systems. So another way will need to be worked out on how to detect of zfs/spl "build". The source is now properly detected.

 

[James A Simmons]

BTW does anyone know if Ubuntu16 creates *.ddeb as well. The *.ddeb are the new debuginfo packages for recent debian/Ubuntu systems.

[James A Simmons]

I'm seeing in the latest testing: 

[96614.006917] osd_ldiskfs*: Unknown symbol ldiskfs_orphan_add (err 0)*

[James A Simmons]

Give the latest patch a try. I worked out all the kinks for ldiskfs. The only thing missing now is a e2fsprogs for Ubuntu18.

[Li Dongyang (Inactive)]

The e2fsprogs is one the way. I have a patched e2fsprogs 1.44.3 working on ubuntu18. Need to clean up and consolidate some commits then we are good.

[James A Simmons]

With the latest patch all you have to do is install the kernel source tree for Ubuntu to generate lustre server packages.

apt-get install linux-source; bunzip2 /usr/src/linux-source-4.15.0.tar.bz2; ...

cd lustre-release;./configure;make debs

Its should automagically build the server packages. To build only client you need to do configure --disable-server. If no linux source tree is available the default behavior is to build a Ubuntu client. 

[James A Simmons]

Can you make public your e2fsprog changes? 

[Li Dongyang (Inactive)]

James,

The latest e2fsprogs is currently at git.whamcloud.com, master-lustre-test branch

https://git.whamcloud.com/?p=tools/e2fsprogs.git;a=shortlog;h=refs/heads/master-lustre-test

[James A Simmons]

I have cloned the repo but  when I run : dpkg-buildpackage -rfakeroot -b

I see: 

make[2]: Entering directory '/home/jsimmons/e2fsprogs/debian/BUILD-STD'

./util/subst -f ./util/subst.conf ../../lib/dirpaths.h.in lib/dirpaths.h

make[2]: ./util/subst: Command not found

Makefile:266: recipe for target 'lib/dirpaths.h' failed

make[2]: *** [lib/dirpaths.h] Error 127

make[2]: Leaving directory '/home/jsimmons/e2fsprogs/debian/BUILD-STD'

Makefile:346: recipe for target 'subs' failed

Sorry I'm a newbie for this.

[Li Dongyang (Inactive)]

There is a wiki page for this:

https://wiki.whamcloud.com/pages/viewpage.action?pageId=63968116#BuildLustreMASTERwithLdiskfs&ZFSonUbuntu14.04/16.04fromGit-4.)(Optional)Buildingmodifiede2fsprogsforLDISKFSsupport

basically what you need to do this clone the repo and checkout the master-lustre-test branch, which is 1.44.3 + lustre related changes.

We don't need to download the debian control tarball from ubuntu and untar/replace the ones in the source tree, they are identical.

Then just follow the remaining steps in the wiki page:

sed -i 's/ext2_types-wrapper.h$//g' lib/ext2fs/Makefile.in

dpkg-buildpackage -b -us -uc

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32613/
Subject: LU-11071 build: Add server build support for Ubuntu 18.04
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e41bdca755954beeb47ee6653ed9e140b7338e96

[James A Simmons]

We still need to figure out ZFS support but ldiskfs is complete.

[Peter Jones]

Including having e2fsprogs support?

[James A Simmons]

Yes we need to get the modified e2fsprogs for this as well. I need to figure out why it doesn't build on Ubuntu18 for me.

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33144
Subject: LU-11071 ldiskfs: fix ldiskfs patch for ubuntu18
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6cbd781c6972cdf4b551f594764801293fc40f13

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33145
Subject: LU-11071 build: use --with-linux-obj for ubuntu
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1b1c4b250b7f4c941a3a9d8e38ad96c52b5ebe54

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33152
Subject: LU-11071 build: create lustre resource package for Ubuntu
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5fd9645febf484d66e2db403b72e7bed7014a6af

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33144/
Subject: LU-11071 ldiskfs: fix ldiskfs patch for ubuntu18
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 072c61ee24b70816721dec1c54d0cc558ea91631

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33152/
Subject: LU-11071 build: create lustre resource package for Ubuntu
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e0aa1c6880f5423adfa9fa024f31990331041c67

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33322
Subject: LU-11071 build: kernel update Ubuntu 18.04
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 738eb6049e95a6e969cd759be0055e62743ac6ce

[Minh Diep]

Hi,

Build e2fsprogs on ubuntu1804 failed with the following

for i in ext2_err.h ext2_types.h ext2_types-wrapper.h; do \
	echo "	INSTALL_DATA /usr/include/ext2fs/$i"; \
	/usr/bin/install -c -m 644 $i /var/lib/jenkins/workspace/sandbox/arch/x86_64/distro/ubuntu1804/debian/tmp/usr/include/ext2fs/$i; \
done
	INSTALL_DATA /usr/include/ext2fs/ext2_err.h
	INSTALL_DATA /usr/include/ext2fs/ext2_types.h
	INSTALL_DATA /usr/include/ext2fs/ext2_types-wrapper.h
/usr/bin/install: cannot stat 'ext2_types-wrapper.h': No such file or directory
Makefile:977: recipe for target 'install' failed
make[2]: *** [install] Error 1
make[2]: Leaving directory '/var/lib/jenkins/workspace/sandbox/arch/x86_64/distro/ubuntu1804/debian/BUILD-STD/lib/ext2fs'
Makefile:429: recipe for target 'install-libs-recursive' failed
make[1]: *** [install-libs-recursive] Error 1
make[1]: Leaving directory '/var/lib/jenkins/workspace/sandbox/arch/x86_64/distro/ubuntu1804/debian/BUILD-STD'
debian/rules:310: recipe for target 'install' failed
make: *** [install] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2

[Dongyang Li]

Hi Minh,

to build e2fsprogs debs on ubuntu, we need to do this:

sed -i 's/ext2_types-wrapper.h$//g' lib/ext2fs/Makefile.in
dpkg-buildpackage -b -us -uc

Also see https://wiki.whamcloud.com/pages/viewpage.action?pageId=63968116#BuildLustreMASTERwithLdiskfs&ZFSonUbuntu14.04/16.04fromGit-4.)(Optional)Buildingmodifiede2fsprogsforLDISKFSsupport

[Andreas Dilger]

Dongyang, do we even need that wrapper anymore? Looking in the RHEL6 .spec file, there is a comment like "remove arch wrapper", so maybe we should also remove it from the THEL6 .spec patch completely?

[Dongyang Li]

We can remove the multilib stuff:

# ugly hack to allow parallel install of 32-bit and 64-bit -devel packages:
%define multilib_arches %{ix86} x86_64 ppc ppc64 s390 s390x sparcv9 sparc64%ifarch %{multilib_arches}
mv -f %{buildroot}%{_includedir}/ext2fs/ext2_types.h \
      %{buildroot}%{_includedir}/ext2fs/ext2_types-%{_arch}.h
mv -f %{buildroot}%{_includedir}/ext2fs/ext2_types-wrapper.h \
      %{buildroot}%{_includedir}/ext2fs/ext2_types.h
%endif

and simply use

# Replace arch-dependent header file with arch-independent stub (when needed).
%multilib_fix_c_header --file %{_includedir}/ext2fs/ext2_types.h

but that means we will add a build dependency of multilib-rpm-config package from epel, which provides the multilib_fix_c_header macro.

Happy to make the change but the build servers need to add epel repo and have multilib-rpm-config installed.

Cheers

DY

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33624
Subject: LU-11071 osd-ldiskfs: support bio integrity with Ubuntu 18
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 25273c05afe93511160fa8e09f730e5b388d80b1

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33626
Subject: LU-11071 build: add files to .gitignore
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 281dcbb842fb0d461cc2451ae5e58e4a51839034

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33322/
Subject: LU-11071 build: kernel update Ubuntu 18.04
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 29029277088c39553081fe3464791b1eb5cf8924

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33624/
Subject: LU-11071 osd-ldiskfs: support bio integrity with Ubuntu 18
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3abacdac1f016a696dd2e791c2b685b94aabe149

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33626/
Subject: LU-11071 build: add files to .gitignore
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b0797d7ed4e6551d229620b45295d9387b3202bd

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33145/
Subject: LU-11071 build: use --with-linux-obj for ubuntu
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f785fcaa7b7fe600bd29efe0ce509beea7ac2095

[Peter Jones]

Landed for 2.12