https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5140 Lustre <p>The Mellanox OFED stack has a compatibility layer to allow it be build across many kernel versions and many distributions. The linux process namespace is one of the things Mellanox creates wrappers to handle various levels of support of this feature. </p> <p>For lustre the libcfs layer also does the same exact thing to handle different levels of support of process namespace. In order to do that libcfs has to figure out which abstract to wrap around, Mellanox or the native system. Currently libcfs doesn't not handle this case properly.</p> Any linux environment that supports process namespace and the external Mellanox OFED stack. In my case it was SLES11 SP3 with a external OFED stack. LU-5140

Mellanox backport header conflicts with newer kernels

Bug Minor Closed Fixed Bob Glossman James A Simmons build Tue, 3 Jun 2014 13:16:07 +0000 Wed, 18 Jun 2014 22:41:46 +0000 Wed, 18 Jun 2014 22:41:46 +0000 Lustre 2.6.0 Lustre 2.6.0 0 6 <p>Here is the fix.</p> <p><a href="http://review.whamcloud.com/#/c/10571" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10571</a></p> <p>I think this ticket should be linked to <a href="https://jira.whamcloud.com/browse/LU-4416" title="support for 3.12 linux kernel" class="issue-link" data-issue-key="LU-4416"><del>LU-4416</del></a>.</p> <p>I ran into the same problem with a recent OFED distro and rhel7. It looks to me like MLNX/OFED chose to keep a local version of uidgid.h to solve this problem. This solution conflicts with the wrappers used in lustre code to address the issue. It's not clear to me how to avoid this conflict. We should adapt correctly to building in environments both with and without any namespace support, with kernel IB stack, with MLNX IB stack, with OFED IB stack.</p> <p>James, you are ahead of me again. I added my comment before I saw your posted solution. How many environments have you tested your fix in? Our build and autotest will only test current distros, which only exercise the case of no namespace support.</p> <p>I ran into this issue using a external Mellanox stack on an SLES11 SP3 system (Cray CLE5.2 environment). I'm going to test the current 2.6 on my test bed Cray system today.</p> <p>Checking back I see that SP3 does have uidgid.h in it too. I didn't notice because it has the relevant CONFIG_ setting turned off by default there, so it always built correctly before we had namespace changes in lustre.</p> <p>Doesn't look like the proposed solution works universally. Builds in our autotest framework are passing, but those are with older OFED versions.</p> <p>I still see failures when trying to build a with a recent OFED; OFED-3.12-rc3; in Centos 6.5. In this case there is no uidgid.h present in kernel source but there is one present in OFED. Due to a lack of one in the kernel, lustre's config.h has #undef HAVE_UIDGID_HEADER. This leads to trying to use local #defines, which in turn leads to compile conflicts with uidgid.h from OFED. example errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> . . . LD [M] /home/bogl/lustre-release/libcfs/libcfs/libcfs.o CC [M] /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.o In file included from /home/bogl/lustre-release/libcfs/include/libcfs/libcfs.h:56, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:79, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:48: error: redefinition of typedef ‘kuid_t’ /usr/src/compat-rdma/include/linux/uidgid.h:50: note: previous declaration of ‘kuid_t’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:49: error: redefinition of typedef ‘kgid_t’ /usr/src/compat-rdma/include/linux/uidgid.h:51: note: previous declaration of ‘kgid_t’ was here In file included from /home/bogl/lustre-release/libcfs/include/libcfs/libcfs.h:56, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:79, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:51:1: error: "INVALID_UID" redefined In file included from /usr/src/compat-rdma/include/linux/compat-2.6.h:19, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:70, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /usr/src/compat-rdma/include/linux/uidgid.h:71:1: error: this is the location of the previous definition In file included from /home/bogl/lustre-release/libcfs/include/libcfs/libcfs.h:56, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:79, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:52:1: error: "INVALID_GID" redefined In file included from /usr/src/compat-rdma/include/linux/compat-2.6.h:19, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:70, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /usr/src/compat-rdma/include/linux/uidgid.h:72:1: error: this is the location of the previous definition In file included from /home/bogl/lustre-release/libcfs/include/libcfs/libcfs.h:56, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:79, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:54:1: error: "GLOBAL_ROOT_UID" redefined In file included from /usr/src/compat-rdma/include/linux/compat-2.6.h:19, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:70, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /usr/src/compat-rdma/include/linux/uidgid.h:68:1: error: this is the location of the previous definition In file included from /home/bogl/lustre-release/libcfs/include/libcfs/libcfs.h:56, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:79, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:55:1: error: "GLOBAL_ROOT_GID" redefined In file included from /usr/src/compat-rdma/include/linux/compat-2.6.h:19, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.h:70, from /home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.c:41: /usr/src/compat-rdma/include/linux/uidgid.h:69:1: error: this is the location of the previous definition /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:65: error: redefinition of ‘__kuid_val’ /usr/src/compat-rdma/include/linux/uidgid.h:53: note: previous definition of ‘__kuid_val’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:70: error: redefinition of ‘__kgid_val’ /usr/src/compat-rdma/include/linux/uidgid.h:58: note: previous definition of ‘__kgid_val’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:75: error: static declaration of ‘backport_make_kuid’ follows non-static declaration /usr/src/compat-rdma/include/linux/uidgid.h:137: note: previous declaration of ‘backport_make_kuid’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:80: error: static declaration of ‘backport_make_kgid’ follows non-static declaration /usr/src/compat-rdma/include/linux/uidgid.h:139: note: previous declaration of ‘backport_make_kgid’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:85: error: static declaration of ‘backport_from_kuid’ follows non-static declaration /usr/src/compat-rdma/include/linux/uidgid.h:142: note: previous declaration of ‘backport_from_kuid’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:90: error: static declaration of ‘backport_from_kgid’ follows non-static declaration /usr/src/compat-rdma/include/linux/uidgid.h:144: note: previous declaration of ‘backport_from_kgid’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:95: error: redefinition of ‘uid_eq’ /usr/src/compat-rdma/include/linux/uidgid.h:74: note: previous definition of ‘uid_eq’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:100: error: redefinition of ‘uid_valid’ /usr/src/compat-rdma/include/linux/uidgid.h:124: note: previous definition of ‘uid_valid’ was here /home/bogl/lustre-release/libcfs/include/libcfs/curproc.h:105: error: redefinition of ‘gid_valid’ /usr/src/compat-rdma/include/linux/uidgid.h:129: note: previous definition of ‘gid_valid’ was here make[7]: *** [/home/bogl/lustre-release/lnet/klnds/o2iblnd/o2iblnd.o] Error 1 make[6]: *** [/home/bogl/lustre-release/lnet/klnds/o2iblnd] Error 2 make[5]: *** [/home/bogl/lustre-release/lnet/klnds] Error 2 make[4]: *** [/home/bogl/lustre-release/lnet] Error 2 make[3]: *** [_module_/home/bogl/lustre-release] Error 2 make[3]: Leaving directory `/home/bogl/rb/BUILD/kernel-2.6.32.431.17.1.l0508' make[2]: *** [modules] Error 2 make[2]: Leaving directory `/home/bogl/lustre-release' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/home/bogl/lustre-release' make: *** [all] Error 2 </pre> </div></div> <p>Looks like we will have to use _LINUX_UIDGID_H as well. Both the linux kernel header and the compact-rdma.h define this to avoid potential conflicts with each other. I don't have a OFED 12 setup right now so try this:</p> <p>diff --git a/libcfs/include/libcfs/curproc.h b/libcfs/include/libcfs/curproc.h<br/> index e5e3d57..aa144e7 100644<br/> &#8212; a/libcfs/include/libcfs/curproc.h<br/> +++ b/libcfs/include/libcfs/curproc.h<br/> @@ -45,6 +45,9 @@</p> <p> #if !defined(HAVE_UIDGID_HEADER) || !defined(_<em>KERNEL</em>_)</p> <p>+#ifndef _LINUX_UIDGID_H<br/> +#define _LINUX_UIDGID_H<br/> +<br/> typedef uid_t kuid_t;<br/> typedef gid_t kgid_t;</p> <p>@@ -106,6 +109,8 @@ static inline bool gid_valid(kgid_t gid)</p> { return (gid != INVALID_GID); } <p>+#endif /* _LINUX_UIDGID_H */<br/> +<br/> #endif</p> <p> int cfs_get_environ(const char *key, char *value, int *val_len);</p> <p>That additional change to curproc.h does repair the build with recent OFED on Centos 6.5, but can't speak to all other variations. Even in the Centos build I'm worried that it may have the effect of using defns from OFED uidgid.h in some places and local defns from curproc.h in others, depending on who includes what exactly.</p> <p>The question becomes the order of importance for the uidgid defines. We have the possible combos of distro, ofed, and libcfs. The order for all code outside of the o2ib LND driver is a no brainier. We use the distro if present and the libcfs if not present. The definitions for OFED don't show up outside the o2ib LND driver. Now in the o2ib LND driver do we want in order of most to least importance:</p> <p>compact-rdma.h -&gt; uidgid.h -&gt; libcfs</p> <p>uidgid.h -&gt; compact-rdma.h -&gt; libcfs</p> <p>As for defining _LINUX_UIDGID_H I really can't see a way around this unless we involve the OFED testing in libcfs autoconf and that would be to messy and ugly.</p> <p>Maloo failed to run for patch <a href="http://review.whamcloud.com/#/c/10571" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10571</a>. Could some one please start the test for this patch. Thank you.</p> <p>James, I've called for a retest. It's back in the queue.</p> <p>When we tried to build master with OFED 3.12, Cray encountered the following build failure, which is also fixed by Jame's patch:</p> <p>[ 140s] CC <span class="error">&#91;M&#93;</span> /usr/src/packages/BUILD/cray-lustre/lnet/klnds/o2iblnd/o2iblnd.o<br/> [ 141s] In file included from /usr/src/packages/BUILD/cray-lustre/libcfs/include/libcfs/linux/linux-prim.h:66,<br/> [ 141s] from /usr/src/packages/BUILD/cray-lustre/libcfs/include/libcfs/linux/libcfs.h:53,<br/> [ 141s] from /usr/src/packages/BUILD/cray-lustre/libcfs/include/libcfs/libcfs.h:47,<br/> [ 141s] from /usr/src/packages/BUILD/cray-lustre/lnet/klnds/o2iblnd/o2iblnd.h:71,<br/> [ 141s] from /usr/src/packages/BUILD/cray-lustre/lnet/klnds/o2iblnd/o2iblnd.c:41:<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:137: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] cc1: warnings being treated as errors<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:137: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:139: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:139: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:142: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:142: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:144: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:144: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:146: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:146: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:148: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:148: error: parameter names (without types) in function declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:152: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:152: error: function declaration isn't a prototype<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:151: error: static declaration of 'LINUX_BACKPORT' follows non-static declaration<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:148: error: previous declaration of 'LINUX_BACKPORT' was here<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h: In function 'LINUX_BACKPORT':<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: 'from_kuid' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: (Each undeclared identifier is reported only once<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: for each function it appears in.)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: 'ns' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: 'uid' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:153: error: called object 'LINUX_BACKPORT(&lt;erroneous-expression&gt;)' is not a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h: At top level:<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:158: error: 'LINUX_BACKPORT' declared as function returning a function<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:158: error: function declaration isn't a prototype<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:157: error: redefinition of 'LINUX_BACKPORT'<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:152: error: previous definition of 'LINUX_BACKPORT' was here<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h: In function 'LINUX_BACKPORT':<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:159: error: 'from_kgid' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:159: error: 'ns' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:159: error: 'gid' undeclared (first use in this function)<br/> [ 141s] /usr/src/kernel-modules-ofed/x86_64/cray_ari_s/include/linux/uidgid.h:159: error: called object 'LINUX_BACKPORT(&lt;erroneous-expression&gt;)' is not a function<br/> [ 141s] make<span class="error">&#91;9&#93;</span>: *** <span class="error">&#91;/usr/src/packages/BUILD/cray-lustre/lnet/klnds/o2iblnd/o2iblnd.o&#93;</span> Error 1<br/> [ 141s] make<span class="error">&#91;8&#93;</span>: *** <span class="error">&#91;/usr/src/packages/BUILD/cray-lustre/lnet/klnds/o2iblnd&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;7&#93;</span>: *** <span class="error">&#91;/usr/src/packages/BUILD/cray-lustre/lnet/klnds&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;6&#93;</span>: *** <span class="error">&#91;/usr/src/packages/BUILD/cray-lustre/lnet&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;_module_/usr/src/packages/BUILD/cray-lustre&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;4&#93;</span>: *** <span class="error">&#91;sub-make&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;3&#93;</span>: *** <span class="error">&#91;all&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;3&#93;</span>: Leaving directory `/usr/src/linux-3.0.101-0.21.1_1.0000.8135-obj/x86_64/cray_ari_s'<br/> [ 141s] make<span class="error">&#91;2&#93;</span>: *** <span class="error">&#91;modules&#93;</span> Error 2<br/> [ 141s] make<span class="error">&#91;2&#93;</span>: Leaving directory `/usr/src/packages/BUILD/cray-lustre'<br/> [ 141s] make<span class="error">&#91;1&#93;</span>: *** <span class="error">&#91;all-recursive&#93;</span> Error 1<br/> [ 141s] make<span class="error">&#91;1&#93;</span>: Leaving directory `/usr/src/packages/BUILD/cray-lustre'<br/> [ 141s] make: *** <span class="error">&#91;all&#93;</span> Error 2<br/> [ 141s] error: Bad exit status from /var/tmp/rpm-tmp.71744 (%build)<br/> [ 141s] <br/> [ 141s] <br/> [ 141s] RPM build errors:<br/> [ 141s] Bad exit status from /var/tmp/rpm-tmp.71744 (%build)</p> <p>Patch has landed. This ticket can be closed.</p> Duplicate LU-5194 Development Rank 1|hzwnnj: Rank (Obsolete) 14179 Severity