[LU-663] Some architectures do not have NUMA features anymore

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.0.0
• Environment: lustre 2.0, x86_64 architectures, recent kernel version

Epic

NUMA performance server

• Assignee: Zhenyu Xu
• Reporter: Gregoire Pichon

Description

Lustre makes some of its working threads (ll_ost_io and ptlrpc_hr) NUMA-aware by starting them spread over the online cpus. These portions of code use the cpu_to_node() and node_to_cpumask() kernel services.

in lustre/ptlrpc/service.c, ptlrpc_main()

#if defined(HAVE_NODE_TO_CPUMASK) && defined(CONFIG_NUMA)
        /* we need to do this before any per-thread allocation is done so that
         * we get the per-thread allocations on local node.  bug 7342 */
        if (svc->srv_cpu_affinity) {
                int cpu, num_cpu;

                for (cpu = 0, num_cpu = 0; cpu < cfs_num_possible_cpus();
                     cpu++) {
                        if (!cfs_cpu_online(cpu))
                                continue;
                        if (num_cpu == thread->t_id % cfs_num_online_cpus())
                                break;
                        num_cpu++;
                }
                cfs_set_cpus_allowed(cfs_current(),
                                     node_to_cpumask(cpu_to_node(cpu)));
        }
#endif

1. The cpu_to_node() service is defined by the kernel either with a #define, or in new kernel version (2.6.24) in some architectures with a symbol export. The symbol export case is not correctly managed by Lustre code and leads to always return 0.

This is because in lustre/include/linux/lustre_compat25.h, we have:

#ifndef cpu_to_node
#define cpu_to_node(cpu)         0
#endif

2. The portions of code that uses the node_to_cpumask() service are protected by the HAVE_NODE_TO_CPUMASK #define. Unfortunately, newer kernel versions do not export this symbol (or its equivalent versions node_to_cpu_mask() and node_2_cpu_mask() ) resulting in having HAVE_NODE_TO_CPUMASK undefined.

I am going to provide a patch through Gerrit for these two issues.

Issue Links

• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,2620
back port to b2_1

[Peter Jones]

Landed for 2.2

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/autoconf/lustre-core.m4