[LU-6228] How to balance network connections across socknal_sd tasks?

• Type: Question/Request
• Resolution: Won't Fix
• Priority: Major
• Labels: sdsc
• Environment: Linux 3.10

Epic/Theme

Performance

• Assignee: Amir Shehata (Inactive)
• Reporter: Rick Wagner (Inactive)

Description

While using the ksocklnd LNET driver, I've noticed uneven load across the socknal_sd* tasks on an OSS. The number of tasks is controllable using combinations of nscheds and cpu_npartitions or cpu_pattern. I've also tried adjusting /proc/sys/lnet/portal_rotor, but this does not appear to be the right thing to try.

On a dual socket, 6 core per processor system with

$ cat ksocklnd.conf 
options ksocklnd nscheds=6 peer_credits=128 credits=1024
$ cat libcfs.conf 
options libcfs cpu_pattern="0[0,1,2,3,4,5] 1[6,7,8,9,10,11]"

there are 12 socknal_sd tasks. However, with up to 60 clients doing the same streaming IO, only 4 of the tasks will be heavily loaded (CPU time over 80%). Oddly, when running an LNET bulk_rw self test, up to 10 of the task will be loaded, and can consume 9.2 GB/s on the server's bonded 40GbE links.

What am I missing? I thought it was the mapping of TCP connections to process, but I can't seem to track them through /proc/*/fd/ and /proc/net/tcp.

I'm working from a recent pull of the master branch.

Issue Links

• is related to: LU-5278 ZFS - many OST watchdogs with IOR