[LU-1398] Problem building fsfilt on 2.1.x branch

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.1, Lustre 2.1.2
• Fix Version/s: Lustre 2.1.2
• Type: Bug
• Priority: Minor
• Reporter: Shuichi Ihara (Inactive)
• Assignee: Minh Diep
• Resolution: Fixed

Environment

CentOS 6.2, Lustre 2.1.1 or b2_1

• Severity: 3

Description

I've been trying to build 2.1.1 on CentOS 6.2 using lbuild, I've tried to avoid doing anything surprising in my setup, the only patches I'm using are to lbuild itself and to the target file to correctly pick up OFED 1.5.4.1. What I'm finding is that the build completes successfully and the RPMs install however I'm unable to load the fsfilt_ldiskfs module due to symbol errors. I'm building from source but not using git so the META file is generated before calling lbuild.

The same process works correctly on the 1.8 branch and on our CentOS 5 build of 2.1.1.

This looks to be the same problem as reported once before on a mailing list but remained unresolved at the time. http://lists.lustre.org/pipermail/lustre-discuss/2010-March/012837.html

Kernel errors when trying to load module:

fsfilt_ldiskfs: no symbol version for ldiskfs_iget
fsfilt_ldiskfs: Unknown symbol ldiskfs_iget
fsfilt_ldiskfs: no symbol version for ldiskfs_free_blocks
fsfilt_ldiskfs: Unknown symbol ldiskfs_free_blocks
fsfilt_ldiskfs: no symbol version for ldiskfs_get_group_desc
fsfilt_ldiskfs: Unknown symbol ldiskfs_get_group_desc
fsfilt_ldiskfs: no symbol version for ext_pblock
fsfilt_ldiskfs: Unknown symbol ext_pblock
fsfilt_ldiskfs: no symbol version for ldiskfs_journal_start_sb
fsfilt_ldiskfs: Unknown symbol ldiskfs_journal_start_sb
fsfilt_ldiskfs: no symbol version for __ldiskfs_journal_stop
fsfilt_ldiskfs: Unknown symbol __ldiskfs_journal_stop
fsfilt_ldiskfs: no symbol version for ldiskfs_force_commit
fsfilt_ldiskfs: Unknown symbol ldiskfs_force_commit
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_store_pblock
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_store_pblock
fsfilt_ldiskfs: no symbol version for ldiskfs_itable_unused_count
fsfilt_ldiskfs: Unknown symbol ldiskfs_itable_unused_count
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_search_right
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_search_right
fsfilt_ldiskfs: no symbol version for ldiskfs_mark_inode_dirty
fsfilt_ldiskfs: Unknown symbol ldiskfs_mark_inode_dirty
fsfilt_ldiskfs: no symbol version for ldiskfs_xattr_get
fsfilt_ldiskfs: Unknown symbol ldiskfs_xattr_get
fsfilt_ldiskfs: no symbol version for ldiskfs_discard_preallocations
fsfilt_ldiskfs: Unknown symbol ldiskfs_discard_preallocations
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_calc_credits_for_insert
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_calc_credits_for_insert
fsfilt_ldiskfs: no symbol version for ldiskfs_xattr_set_handle
fsfilt_ldiskfs: Unknown symbol ldiskfs_xattr_set_handle
fsfilt_ldiskfs: no symbol version for ldiskfs_read_inode_bitmap
fsfilt_ldiskfs: Unknown symbol ldiskfs_read_inode_bitmap
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_insert_extent
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_insert_extent
fsfilt_ldiskfs: no symbol version for __ldiskfs_journal_get_write_access
fsfilt_ldiskfs: Unknown symbol __ldiskfs_journal_get_write_access
fsfilt_ldiskfs: no symbol version for ldiskfs_mb_new_blocks
fsfilt_ldiskfs: Unknown symbol ldiskfs_mb_new_blocks
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_search_left
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_search_left
fsfilt_ldiskfs: no symbol version for ldiskfs_bread
fsfilt_ldiskfs: Unknown symbol ldiskfs_bread
fsfilt_ldiskfs: no symbol version for ldiskfs_ext_walk_space
fsfilt_ldiskfs: Unknown symbol ldiskfs_ext_walk_space
fsfilt_ldiskfs: no symbol version for ldiskfs_map_inode_page
fsfilt_ldiskfs: Unknown symbol ldiskfs_map_inode_page
fsfilt_ldiskfs: no symbol version for __ldiskfs_handle_dirty_metadata
fsfilt_ldiskfs: Unknown symbol __ldiskfs_handle_dirty_metadata

Some output from the build process:

I note that the first error about not being able to find fsfilt-ldiskfs.c is present in the Whamcloud build logs in Jenkins so I assume that problem is benign.

CC: gcc
LD:
CPPFLAGS: -include /var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/config.h -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/libcfs/include -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lnet/include -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/include
LLCPPFLAGS: -D_arch_lib_ -D_LARGEFILE64_SOURCE=1
CFLAGS: -g -O2 -Werror -Werror
EXTRA_KCFLAGS: -include /var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/config.h -g -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/libcfs/include -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lnet/include -I/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/include
LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE

Type 'make' to build Lustre.
+ make -j 1 -s
Making all in .
WARNING: could not open /var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt-ldiskfs.c: No such file or directory
WARNING: "__ldiskfs_handle_dirty_metadata" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_bread" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_add_dot_dotdot" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_journal_get_write_access" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_append" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_delete_entry" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_add_entry" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_find_entry" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_mark_inode_dirty" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_force_commit" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "dynlock_lock" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_std_error" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "dynlock_unlock" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_create_inode" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_journal_stop" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfsfs_dirhash" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_journal_start_sb" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "ldiskfs_iget" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/osd-ldiskfs/osd_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_handle_dirty_metadata" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_map_inode_page" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_walk_space" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_bread" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_search_left" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_mb_new_blocks" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_journal_get_write_access" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_insert_extent" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_read_inode_bitmap" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_xattr_set_handle" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_calc_credits_for_insert" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_discard_preallocations" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_xattr_get" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_mark_inode_dirty" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_search_right" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_itable_unused_count" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_ext_store_pblock" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_force_commit" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "__ldiskfs_journal_stop" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_journal_start_sb" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ext_pblock" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_get_group_desc" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_free_blocks" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
WARNING: "ldiskfs_iget" [/var/tmp/lustre_build_CtC/BUILD/lustre-2.1.1/lustre/lvfs/fsfilt_ldiskfs.ko] undefined!
Making all in libsysio
Making all in lustre-iokit
Making all in obdfilter-survey
Making all in sgpdd-survey
Making all in ost-survey

I can supply full build logs if required.

Comments

[Peter Jones]

Minh

Could you please look into this one?

Thanks

Peter

[Andreas Dilger]

Ashley, do you have the ldiskfs module loaded? It looks like the missing symbols are from ldiskfs.ko.

[Ashley Pittman (Inactive)]

Yes, I can load the ldiskfs module but not fsfilt_ldiskfs. nm is showing ldiskfs as exporting the symbols required as well.

[root@12k-1 2.6.32-220.13.1.el6_lustre20120511113541.x86_64]# modinfo ldiskfs
filename: /lib/modules/2.6.32-220.13.1.el6_lustre20120511113541.x86_64/updates/kernel/fs/lustre-ldiskfs/ldiskfs.ko
license: GPL
description: Fourth Extended Filesystem
author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others
srcversion: BDFD85813EE5F5891ACB6BB
depends: mbcache,jbd2
vermagic: 2.6.32-220.13.1.el6_lustre20120511113541.x86_64 SMP mod_unload modversions
[root@12k-1 2.6.32-220.13.1.el6_lustre20120511113541.x86_64]# nm /lib/modules/2.6.32-220.13.1.el6_lustre20120511113541.x86_64/updates/kernel/fs/lustre-ldiskfs/ldiskfs.ko | grep ldiskfs_iget
00000000dd947564 A __crc_ldiskfs_iget
0000000000000080 r __kcrctab_ldiskfs_iget
000000000000018d r __kstrtab_ldiskfs_iget
0000000000000100 r __ksymtab_ldiskfs_iget
0000000000014a10 T ldiskfs_iget

[Ashley Pittman (Inactive)]

I've been investigating this further, it looks like the build server is running out of disk space during the build, there are no errors reported and the build still completes but the free space does reach zero. Let me rebuild this VM with more space and see if the problem persists.

[Ashley Pittman (Inactive)]

I can confirm it's not the disk space, I've doubled the size from 16G to 32G and the get the same results.

[Ashley Pittman (Inactive)]

I did some more experimenting over the weekend and this looks to be more common than I thought.

If I switch back from external OFED to inkernel ofed the build succeeds and I can load the module and bring up a filesystem. As a test I switched our 1.8.7 build from inkernel to external OFED and that then failed so this is related to how OFED is picked.

I've found a couple of other interesting datapoints as well, looking in Jenkins at the build logs for l2_1 on el5 (it's not build with external OFED on el6) I see the same build errors in the logs about symbol errors for the "ofa" build but not for the inkernel build, compare these two build logs, search for "WARNING" to see the difference.

http://build.whamcloud.com/job/lustre-b2_1/arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/lastBuild/consoleText
http://build.whamcloud.com/job/lustre-b2_1/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/lastBuild/consoleText

Finally, I wonder if this is an issue that's been present for a while in the coebase but has recently got worse, if I look at a stock 1.8.7 install using the Whamcloud RPMs there is an warning in the kernel logs about fsfilt_ldisksfs

[root@mini-oss1 ~]# cat /proc/fs/lustre/version
lustre: 1.8.6
kernel: patchless_client
build: jenkins-wc1--PRISTINE-2.6.18-238.12.1.el5_lustre.gce5e033
[root@mini-oss1 ~]# dmesg | grep fsfilt
fsfilt_ldiskfs: no version for "ldiskfs_iget" found: kernel tainted.

[Andreas Dilger]

The "kernel tainted" message should not affect module functionality, unless the newer kernels have some kind of extra checking to prevent inserting modules for some reason.

As for the OFED issues, that is outside my experience, so I can't comment.

[Minh Diep]

Ok, I can reproduce this in our lab. I am investigating

[Minh Diep]

Hi Ashley,

I have tried to build 2.1 with ofed 1.5.4, 1.5.3.1, 1.5.3 and hit the same issue.
I did not see this issue in lustre 2.2. Could you give it a try while I am continue to find the cause?

[Minh Diep]

the patch for this problem is: http://review.whamcloud.com/#change,2898

Ashley, Please try and let me know

Thanks

[Ashley Pittman (Inactive)]

Yes, that fixes the problem for us. We now get no errors when loading the module, not even the tainted kernel message.

[Peter Jones]

Landed for 2.1.2