[LU-1164] ko2iblnd schedulers

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.1.3
• Type: Improvement
• Priority: Minor
• Reporter: Sebastien Buisson (Inactive)
• Assignee: Liang Zhen (Inactive)
• Resolution: Fixed

Description

Hi,

We believe that it could be a nice feature to be able control the number of kib scheduler threads launched by the ko2iblnd module.
Indeed, we can have some situations where the default threads number is not appropriate. In the current code it creates as many threads as the number of CPU cores, which means 32 or even 40 on our MESCA platform. We have noticed a performance penalty when running so many kib scheduler threads.

This is why I propose a patch that gives the ability to choose the number of kib scheduler threads. This is done via a ko2iblnd kernel module option. The default value remains unchanged.

Cheers,
Sebastien.

Comments

[Liang Zhen (Inactive)]

Sebastien, yes indeed, I've put this on my radar for a while, we created way too many LND threads especially on those machines with tens of cores and HT. I will have a long term solution in 2.3, and will consider about have a short term fix for this.

[Sebastien Buisson (Inactive)]

Hi,

I have uploaded the patch at:

http://review.whamcloud.com/2246

[Andreas Dilger]

Sebastien,
thank you for the patch. Having a tunable is a good start, but even better would be if this is auto tuning/limiting in some way. When there are so many cores, is it enough to have one thread per socket? One or two threads per interface?

By having the default values be better, and a tunable to override this value, it is easier for users to get a good experience with Lustre out of the box, instead of having to tune dozens of parameters.

[Liang Zhen (Inactive)]

Andreas, I will have a long term fix for this in project Apus (SMP node affinity)

[Sebastien Buisson (Inactive)]

Hi,

As Liang explained, the same thing has been done in a large set of patches bringing SMP scalability to master. So I understand it will not be possible to port it to former branches.

For that purpose, I have created a b2_1 version of my patch accessible here:
http://review.whamcloud.com/#change,3047
and also a b2_2 version here:
http://review.whamcloud.com/#change,3044

It it possible to have them reviewed?

Thanks,
Sebastien.

[Liang Zhen (Inactive)]

Patch is in 2.1, 2,3 and later have different solution