[LU-663] Some architectures do not have NUMA features anymore

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.0.0
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.2
• Type: Improvement
• Priority: Minor
• Reporter: Gregoire Pichon
• Assignee: Zhenyu Xu
• Resolution: Fixed

Environment

lustre 2.0, x86_64 architectures, recent kernel version

• Epic: NUMA, performance, server

Description

Lustre makes some of its working threads (ll_ost_io and ptlrpc_hr) NUMA-aware by starting them spread over the online cpus. These portions of code use the cpu_to_node() and node_to_cpumask() kernel services.

in lustre/ptlrpc/service.c, ptlrpc_main()

#if defined(HAVE_NODE_TO_CPUMASK) && defined(CONFIG_NUMA)
        /* we need to do this before any per-thread allocation is done so that
         * we get the per-thread allocations on local node.  bug 7342 */
        if (svc->srv_cpu_affinity) {
                int cpu, num_cpu;

                for (cpu = 0, num_cpu = 0; cpu < cfs_num_possible_cpus();
                     cpu++) {
                        if (!cfs_cpu_online(cpu))
                                continue;
                        if (num_cpu == thread->t_id % cfs_num_online_cpus())
                                break;
                        num_cpu++;
                }
                cfs_set_cpus_allowed(cfs_current(),
                                     node_to_cpumask(cpu_to_node(cpu)));
        }
#endif

1. The cpu_to_node() service is defined by the kernel either with a #define, or in new kernel version (2.6.24) in some architectures with a symbol export. The symbol export case is not correctly managed by Lustre code and leads to always return 0.

This is because in lustre/include/linux/lustre_compat25.h, we have:

#ifndef cpu_to_node
#define cpu_to_node(cpu)         0
#endif

2. The portions of code that uses the node_to_cpumask() service are protected by the HAVE_NODE_TO_CPUMASK #define. Unfortunately, newer kernel versions do not export this symbol (or its equivalent versions node_to_cpu_mask() and node_2_cpu_mask() ) resulting in having HAVE_NODE_TO_CPUMASK undefined.

I am going to provide a patch through Gerrit for these two issues.

Comments

[Peter Jones]

Bobi

Can you please look into this one?

Thanks

Peter

[Gregoire Pichon]

I have pushed a patch in Gerrit:
http://review.whamcloud.com/1345

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/autoconf/lustre-core.m4
• lustre/ptlrpc/service.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• libcfs/include/libcfs/linux/linux-prim.h
• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,sles11,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• libcfs/include/libcfs/linux/linux-prim.h
• lustre/ptlrpc/service.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• libcfs/include/libcfs/linux/linux-prim.h
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #374
LU-663 kernel: Some arch do not have NUMA features anymore (Revision 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8)

Result = SUCCESS
Oleg Drokin : 49b74e8ab7c0bf76a4ff5b0b2cce15314b1ca8f8
Files :

• lustre/ptlrpc/service.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• libcfs/include/libcfs/linux/linux-prim.h

[Peter Jones]

Landed for 2.2

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,2620
back port to b2_1