Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-2207] planning upgrade to 2.3 to increase metadata performance using smp affinity. https://jira.whamcloud.com/browse/LU-2207 Lustre We are planning to upgrade to the 2.3 lustre version to increase metadata performance expecially using the smp affinity. We have a Intel Nehalem 2 socket, 4 cores each socket as a mds and Intel Sandy Bridge 2 socket, 8 cores each. LU-2207

planning upgrade to 2.3 to increase metadata performance using smp affinity.

Bug Minor Resolved Not a Bug Liang Zhen Supporto Lustre Jnet2000 Wed, 17 Oct 2012 20:34:15 +0000 Wed, 24 Oct 2012 17:16:30 +0000 Wed, 24 Oct 2012 17:16:30 +0000 Lustre 2.3.0 0 3 Hi, could you help us to planning the configuration of the lustre mds and the lustre client in term of Compute Partitions to maximize the metadata performance? Thanks in advance. Every node has 1 infiniband hca (ib0). Hi there When is your timeframe for this upgrade? We are still in final release testing for 2.3 and it is not GA yet... Peter Hi Peter, we are planning for the end of the year,but the customer need a detailed plan for the next week to approve the project. I yust need more information on the configuration of "smp affinity" for our hardware because the information that I find in the <a href="https://jira.whamcloud.com/browse/LU-56" title="Finish SMP scalability work" class="issue-link" data-issue-key="LU-56"><del>LU-56</del></a> is not enough to understand well. Could you help me? Thanks in advance. ok. I am assigning this ticket to the engineer who worked on this feature. Could you please elaborate with some specifics as to what you need to know? If any action is needed then I would expect this to be covered in the latest version of the Lustre manual. Our goal is to increase the metadata performance. We have a Intel Nehalem 2 socket, 4 cores each socket as a mds and Intel Sandy Bridge 2 socket, 8 cores each as client. Our network is a QDR infiniband. this is going to be covered by Lustre manual very soon. A few questions: <ul> <li>is there router in your cluster?</li> <li>how many clients node in the cluster?</li> <li>how many OSS and OSTs in the cluster? But I think it's worth a try with all defaults, of course, there are a few things can be tuned:</li> <li>turne on "Portal Rotor" on MDS, it can be tuned on by "echo on > /proc/sys/lnet/portal_rotor" (please check define or "portal rotor" from here: <a href="http://jira.whamcloud.com/secure/attachment/11805/opensfs_smp_demonstration.pdf" class="external-link" rel="nofollow">http://jira.whamcloud.com/secure/attachment/11805/opensfs_smp_demonstration.pdf</a>)</li> <li>By default Lustre will create 2 CPU partitions on 2X4 cores MDS, you can try to set it to 4 (options libcfs cpu_npartitions=4) and choose the better one.</li> </ul> I've setting up a small lab with 6 clients (Sandy Bridge Intel CPU) and 2 server (Nehalem Intel CPU). I'm trying with lustre 2.3RC2. I respond to your question for the lab that I have setting up. 1) no router yust a mellanox switch, we use infiniband connections 2) 6 clients 3) 2 server. I have 4 lun for the OSS and 1 lun for the MDS. I plan to use 1 server as MDS and 1 server as OSS. 4) okay 5) I see this in the /var/log/message: Oct 20 09:25:37 gemini2 kernel: LNet: HW CPU cores: 8, npartitions: 2 Oct 20 09:25:37 gemini2 kernel: alg: No test for crc32 (crc32-table) Oct 20 09:25:37 gemini2 kernel: alg: No test for adler32 (adler32-zlib) Oct 20 09:25:41 gemini2 kernel: padlock: VIA PadLock Hash Engine not detected. Oct 20 09:25:41 gemini2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-279.5.1.el6_lustre.g8932d49.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device Oct 20 09:25:50 gemini2 kernel: Lustre: Lustre: Build Version: 2.3.0-RC2--PRISTINE-2.6.32-279.5.1.el6_lustre.g8932d49.x86_64 Oct 20 09:25:50 gemini2 kernel: LNet: Added LNI 192.168.3.2@o2ib [8/256/0/180] gemini1 and gemini2 are the servers and this on the clients: Oct 20 09:57:53 virgo1 kernel: LNet: HW CPU cores: 16, npartitions: 4 Oct 20 09:57:53 virgo1 kernel: alg: No test for crc32 (crc32-table) Oct 20 09:57:53 virgo1 kernel: alg: No test for adler32 (adler32-zlib) Oct 20 09:57:53 virgo1 kernel: alg: No test for crc32 (crc32-pclmul) Oct 20 09:57:57 virgo1 kernel: padlock: VIA PadLock Hash Engine not detected. Oct 20 09:57:57 virgo1 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.32-279.5.1.el6.x86_64/kernel/drivers/crypto/padlock-sha.ko): No such device Oct 20 09:58:06 virgo1 kernel: Lustre: Lustre: Build Version: 2.3.0-RC2--PRISTINE-2.6.32-279.5.1.el6.x86_64 Oct 20 09:58:06 virgo1 kernel: LNet: Added LNI 192.168.3.14@o2ib [8/256/0/180] yes, these should be fine, there are two concerns: <ul> <li>I'm not sure 6 clients can generate enough load on MDS, because SMP feature is supposed to improve aggregation performance for server side, not for each single client</li> <li>1 OSS is probably not very enough, because load on OSS could be high, especially if file stripecount > 1.</li> </ul> But it's worth a try to see what you can get. Another suggestion is: please use big journal size, for example: 8G external journal for MDS(it could be better to put journal on a fast device like SSD), or at least 2G internal journal for both MDS and OSS. Hi, I have a 100GB SSD on the MDS. Thanks. Hi, I try to test without the smp affinity. To disable this feature is sufficient to put in the modprobe.conf "options libcfs cpu_npartitions=0" ? Thanks well, we can't totally disable SMP feature on 2.3, because there are many locking & threading improvements even CPU partition is disabled. So the only way to test w/o SMP feature is running 2.2 or 2.1, but still, there will be some "noise" because there are many other changes in 2.3. Thank you. Could you close this issue? ok thanks! Development Rank 1|hzvalj: Rank (Obsolete) 5252 Severity