[LU-2207] planning upgrade to 2.3 to increase metadata performance using smp affinity.

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.3.0
• Type: Bug
• Priority: Minor
• Reporter: Supporto Lustre Jnet2000 (Inactive)
• Assignee: Liang Zhen (Inactive)
• Resolution: Not a Bug

Environment

We are planning to upgrade to the 2.3 lustre version to increase metadata performance expecially using the smp affinity. We have a Intel Nehalem 2 socket, 4 cores each socket as a mds and Intel Sandy Bridge 2 socket, 8 cores each.

• Severity: 3

Comments

[Supporto Lustre Jnet2000 (Inactive)]

Hi,
could you help us to planning the configuration of the lustre mds and the lustre client in term of Compute Partitions to maximize the metadata performance? Thanks in advance.

[Supporto Lustre Jnet2000 (Inactive)]

Every node has 1 infiniband hca (ib0).

[Peter Jones]

Hi there

When is your timeframe for this upgrade? We are still in final release testing for 2.3 and it is not GA yet...

Peter

[Supporto Lustre Jnet2000 (Inactive)]

Hi Peter,
we are planning for the end of the year,but the customer need a detailed plan for the next week to approve the project. I yust need more information on the configuration of "smp affinity" for our hardware because the information that I find in the LU-56 is not enough to understand well. Could you help me?

Thanks in advance.

[Peter Jones]

ok. I am assigning this ticket to the engineer who worked on this feature. Could you please elaborate with some specifics as to what you need to know? If any action is needed then I would expect this to be covered in the latest version of the Lustre manual.

[Supporto Lustre Jnet2000 (Inactive)]

Our goal is to increase the metadata performance. We have a Intel Nehalem 2 socket, 4 cores each socket as a mds and Intel Sandy Bridge 2 socket, 8 cores each as client. Our network is a QDR infiniband.

[Liang Zhen (Inactive)]

this is going to be covered by Lustre manual very soon.
A few questions:

• is there router in your cluster?
• how many clients node in the cluster?
• how many OSS and OSTs in the cluster?
  But I think it's worth a try with all defaults, of course, there are a few things can be tuned:
• turne on "Portal Rotor" on MDS, it can be tuned on by "echo on > /proc/sys/lnet/portal_rotor" (please check define or "portal rotor" from here: http://jira.whamcloud.com/secure/attachment/11805/opensfs_smp_demonstration.pdf)
• By default Lustre will create 2 CPU partitions on 2X4 cores MDS, you can try to set it to 4 (options libcfs cpu_npartitions=4) and choose the better one.

[Supporto Lustre Jnet2000 (Inactive)]

I've setting up a small lab with 6 clients (Sandy Bridge Intel CPU) and 2 server (Nehalem Intel CPU).
I'm trying with lustre 2.3RC2.

I respond to your question for the lab that I have setting up.

1) no router yust a mellanox switch, we use infiniband connections
2) 6 clients
3) 2 server. I have 4 lun for the OSS and 1 lun for the MDS. I plan to use 1 server as MDS and 1 server as OSS.
4) okay
5) I see this in the /var/log/message:

Oct 20 09:25:37 gemini2 kernel: LNet: HW CPU cores: 8, npartitions: 2
Oct 20 09:25:37 gemini2 kernel: alg: No test for crc32 (crc32-table)
Oct 20 09:25:37 gemini2 kernel: alg: No test for adler32 (adler32-zlib)
Oct 20 09:25:41 gemini2 kernel: padlock: VIA PadLock Hash Engine not detected.
Oct 20 09:25:41 gemini2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-279.5.1.el6_lustre.g8932d49.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Oct 20 09:25:50 gemini2 kernel: Lustre: Lustre: Build Version: 2.3.0-RC2--PRISTINE-2.6.32-279.5.1.el6_lustre.g8932d49.x86_64
Oct 20 09:25:50 gemini2 kernel: LNet: Added LNI 192.168.3.2@o2ib [8/256/0/180]

gemini1 and gemini2 are the servers

and this on the clients:

Oct 20 09:57:53 virgo1 kernel: LNet: HW CPU cores: 16, npartitions: 4
Oct 20 09:57:53 virgo1 kernel: alg: No test for crc32 (crc32-table)
Oct 20 09:57:53 virgo1 kernel: alg: No test for adler32 (adler32-zlib)
Oct 20 09:57:53 virgo1 kernel: alg: No test for crc32 (crc32-pclmul)
Oct 20 09:57:57 virgo1 kernel: padlock: VIA PadLock Hash Engine not detected.
Oct 20 09:57:57 virgo1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-279.5.1.el6.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device
Oct 20 09:58:06 virgo1 kernel: Lustre: Lustre: Build Version: 2.3.0-RC2--PRISTINE-2.6.32-279.5.1.el6.x86_64
Oct 20 09:58:06 virgo1 kernel: LNet: Added LNI 192.168.3.14@o2ib [8/256/0/180]

[Liang Zhen (Inactive)]

yes, these should be fine, there are two concerns:

• I'm not sure 6 clients can generate enough load on MDS, because SMP feature is supposed to improve aggregation performance for server side, not for each single client
• 1 OSS is probably not very enough, because load on OSS could be high, especially if file stripecount > 1.

But it's worth a try to see what you can get.
Another suggestion is: please use big journal size, for example: 8G external journal for MDS(it could be better to put journal on a fast device like SSD), or at least 2G internal journal for both MDS and OSS.

[Supporto Lustre Jnet2000 (Inactive)]

Hi,
I have a 100GB SSD on the MDS. Thanks.

[Supporto Lustre Jnet2000 (Inactive)]

Hi,
I try to test without the smp affinity. To disable this feature is sufficient to put in the modprobe.conf "options libcfs cpu_npartitions=0" ?

Thanks

[Liang Zhen (Inactive)]

well, we can't totally disable SMP feature on 2.3, because there are many locking & threading improvements even CPU partition is disabled. So the only way to test w/o SMP feature is running 2.2 or 2.1, but still, there will be some "noise" because there are many other changes in 2.3.

[Supporto Lustre Jnet2000 (Inactive)]

Thank you. Could you close this issue?

[Peter Jones]

ok thanks!