[LU-1144] implement a NUMA aware ptlrpcd binding policy

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.3.0
• Type: Improvement
• Priority: Minor
• Reporter: Gregoire Pichon
• Assignee: Andreas Dilger
• Resolution: Fixed

Environment

lustre 2.1 with patch ORNL-22, machine with several NUMA nodes

Description

The binding policy implemented by the ORNL-22 patch does not take care of NUMA aspects. It means ptlrpcd thread partners can be in different NUMA nodes. When the io requests will be processed by ptlrpcd threads in their affinity domain it will be more performant to have ptlrpcd partners in the same NUMA domain.

Comments

[Gregoire Pichon]

I have uploaded the patch into Gerrit: http://review.whamcloud.com/2212

[Peter Jones]

Andreas has been inspecting this patch

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,sles11,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,ofa #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,inkernel #483
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el5,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el6,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-dev » i686,server,el5,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el6,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el5,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/ptlrpc/ptlrpcd.c
• lustre/include/lustre_net.h

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el5,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el6,inkernel #340
LU-1144 ptlrpc: implement a NUMA aware ptlrpcd binding policy (Revision db8a6829cb1c9869d998920c095a14de2d5300bf)

Result = SUCCESS
Oleg Drokin : db8a6829cb1c9869d998920c095a14de2d5300bf
Files :

• lustre/include/lustre_net.h
• lustre/ptlrpc/ptlrpcd.c

[Peter Jones]

Landed for 2.3

[Jay Lan (Inactive)]

Not sure if this "NUMA aware" patch helps Lustre BZ-23047 "zone_reclaim_mode = 1 tanks lustre performance on clients", so please advise. Appreciated!

My understanding of the BZ-23047 problem was that zone_reclaim tried to free cache memory aggressively but failed since Lustre held a lock of those clean cache. Without freeing those clean cache, memory allocation may have to grab from nodes of some distance.