https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12667 Lustre <p>For instance, AMD EPYC 7551(32 CPU cores) has 4 dies on an CPU socket and each die consists of 8 CPU cores and also each numa node.<br/> If two CPU sockets per client, total 64 CPU cores (128 CPU cores with logical processors) and 8 NUMA nodes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># numactl -H available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 4 5 6 7 64 65 66 67 68 69 70 71 node 0 size: 32673 MB node 0 free: 31561 MB node 1 cpus: 8 9 10 11 12 13 14 15 72 73 74 75 76 77 78 79 node 1 size: 32767 MB node 1 free: 31930 MB node 2 cpus: 16 17 18 19 20 21 22 23 80 81 82 83 84 85 86 87 node 2 size: 32767 MB node 2 free: 31792 MB node 3 cpus: 24 25 26 27 28 29 30 31 88 89 90 91 92 93 94 95 node 3 size: 32767 MB node 3 free: 31894 MB node 4 cpus: 32 33 34 35 36 37 38 39 96 97 98 99 100 101 102 103 node 4 size: 32767 MB node 4 free: 31892 MB node 5 cpus: 40 41 42 43 44 45 46 47 104 105 106 107 108 109 110 111 node 5 size: 32767 MB node 5 free: 30676 MB node 6 cpus: 48 49 50 51 52 53 54 55 112 113 114 115 116 117 118 119 node 6 size: 32767 MB node 6 free: 30686 MB node 7 cpus: 56 57 58 59 60 61 62 63 120 121 122 123 124 125 126 127 node 7 size: 32767 MB node 7 free: 32000 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 </pre> </div></div> <p>Also first generation EYPC(Naples) has PCI controller per die (NUMA node) and IB HCA connected to one of PCIe controller like below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /sys/class/infiniband/mlx5_0/device/numa_node 5 </pre> </div></div> <p>mlx5_0 adapter conneted to CPU1's NUMA node1 which is numa node 5 in 2 socket configuration.</p> <p>In this case, default LNET performance doesn't perform well and it requires manual CPT setting, but still highly relies on what CPT configuration and cpu cores are involved.<br/> Here is quick LNET selftest results with default CPT and NUMA aware an CPT configuration.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>default CPT setting(cpu_npartitions=8) client:server PUT(GB/s) GET(GB/s) 1:1 7.0 6.8 1:2 11.3 3.2 1:4 11.4 3.4 1 CPT(cpu_npartitions=1 cpu_pattern="0[40-47,104,111]") client:server PUT(GB/s) GET(GB/s) 1:1 11.0 11.0 1:2 11.4 11.4 1:4 11.4 11.4 </pre> </div></div> <p>numa aware CPT configuration made much better LNET performance, but CPT requires not only LNET, but also all other Lustre client threads. In general, we want to get more number of CPU cores and CPTs involved, but LNET needs to be ware of CPT and NUMA node where network interface installed.</p> master branch, AMD EYPC CPU LU-12667

Read doesn't perform well in complex NUMA configuration

Bug Major Open Unresolved WC Triage Shuichi Ihara Thu, 15 Aug 2019 14:19:41 +0000 Mon, 8 Jun 2020 22:19:40 +0000 0 13 <p>Then, assign of an CPT to NI works. automated fine tunning (e.g defects numa node for NI) might be better, but manual setting is workable solsuion as a workaround.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options lnet networks="o2ib10(ib0)[5]" # cat /sys/kernel/debug/lnet/cpu_partition_table 0 : 0 1 2 3 4 5 6 7 64 65 66 67 68 69 70 71 1 : 8 9 10 11 12 13 14 15 72 73 74 75 76 77 78 79 2 : 16 17 18 19 20 21 22 23 80 81 82 83 84 85 86 87 3 : 24 25 26 27 28 29 30 31 88 89 90 91 92 93 94 95 4 : 32 33 34 35 36 37 38 39 96 97 98 99 100 101 102 103 5 : 40 41 42 43 44 45 46 47 104 105 106 107 108 109 110 111 6 : 48 49 50 51 52 53 54 55 112 113 114 115 116 117 118 119 7 : 56 57 58 59 60 61 62 63 120 121 122 123 124 125 126 127 </pre> </div></div> <p>For the sake of keeping a record. As discussed:</p> <p>the performance issue comes from the fact that the LND threads are spread across all the CPTs<br/> I'm guessing in this NUMA configuration that has a performance impact<br/> by restricting the NI on the set of CPTs you're interested in, then the LND threads are only spawned on these cores. <br/> RDMA is more efficient since it doesn't have to cross NUMA boundary.</p> <p>The issue with automated tuning is that there is no criteria to base the tuning on in this case. Do you have suggestion on how to automate this config?</p> <p>Originally, we wanted good single client write and read performance, but initial read performance was bad. I was thinking this was because cpt configuration for LNET was not optimal for such numa node configuration. However, problem seems to be even complex and still not sure this is LNET problem or others.</p> <p>Here is quick test results.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /etc/modprobe.d/lustre.conf options lnet networks="o2ib10(ib0)[5]" </pre> </div></div> <p>LNET selftest (RPC PUT)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client - oss1 11.0GB/sec client - oss2 11.0GB/sec client - oss1,oss2 (distributed) 11.4GB/sec </pre> </div></div> <p>All lnet performance seems to be good.</p> <p>IOR (read, FPP, 1MB)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mpirun -np 32 /work/tools/bin/ior -o /scratch0/2ost/file -a POSIX -r -e -b 4g -t 1m -F -C -vv -k client - oss1 (2xOST) 8.1GB/sec client - oss2 (2xOST) 8.1GB/sec client - oss1,oss2 (4xOST) distributed 3.9GB/sec client - oss1 (4xOST) 8.1GB/sec </pre> </div></div> <p>If client read data from single OSS, performance is reasonable (still not perfect, but not so bad), but if client talks multiple OSSs, read performance drops.<br/> I thought it's related to numbers of OSC, but as far as I tested same same 4xOSC, but it was single OSS, performance is good. I will dig more..</p> <p>Well, that makes sense to me if it's a CPT binding issue of some kind, because the CPT binding is linked to the OSS, not OST.  And the CPT binding stuff in Lustre on the client mostly matters at the Lnet/o2ib type layers, as you know, so...  That sort of fits.</p> <p>Hm.  I'll reply to your email.</p> <p>it's same read preformance degradation regardless CPT binding or not.<br/> But, at least, I saw good lnet selftest performance even against multiple OSSs with CPT binding, but when client does actual IO read operation, perforamnce doesn't scale by nunber of OSS.</p> <p>FYI, there is a known problem for read for striped files:</p> <p><a href="https://review.whamcloud.com/#/c/35438/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35438/</a></p> <p>This should help read for striped different OST/OSS performances I guess.</p> <p>this is not striped file and it's file-per-process.. </p> <p>One thing to consider is that when RDMAing to/from buffers, these buffers are allocated on a specific NUMA node. They could be spread across all the NUMA nodes. If the NUMA node the buffer is allocated on is far from the IB interface doing the RDMAing, then it would impact performance. When we were doing the MR testing we noticed a significant impact due to these NUMA penalties. Granted they were on large UV machine, but the same problem could be happening here as well.</p> <p>One thing to try is to restrict buffer allocation to NUMA node 5. Can we try this and see how it impacts performance.</p> <p>OK, here is a test result which configured only an CPT and allocates all cpus in numa node 5 into that CPT.<br/> like this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options lnet networks="o2ib10(ib0)" options libcfs cpu_npartitions=1 cpu_pattern="0[40-47,104,111]" # cat /sys/kernel/debug/lnet/cpu_partition_table 0 : 40 41 42 43 44 45 46 47 104 111 </pre> </div></div> <p>LNET selftest performs 11GB/sec from single client to against either 2 or 4 servers, but IOR read still hit 4GB/sec against 2 OSS (not only two OSS, but also any number of multiple servers)<br/> If number of OSS reduced to 1, performance goes up 8GB/sec. This is exact same IOR results above.</p> <p>If another interface is added on client as multi-rail, read performance bumps up.<br/> but, it's a bit strange.. if I added an interface which is same numa node as primary interface, performance doesn't scale. but if i added an interface which is different numa node from primary interface, performance improved.</p> <p>e.g. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@mds15:~# cat /sys/class/net/ib0/device/numa_node 5 root@mds15:~# cat /sys/class/net/ib1/device/numa_node 5 root@mds15:~# cat /sys/class/net/ib2/device/numa_node 6 root@mds15:~# cat /sys/class/net/ib3/device/numa_node 6 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options lnet networks="o2ib10(ib0)" Max Read: 3881.91 MiB/sec (4070.48 MB/sec) options lnet networks="o2ib10(ib0,ib1)" Max Read: 3193.72 MiB/sec (3348.86 MB/sec) options lnet networks="o2ib10(ib0,ib2)" Max Read: 6110.81 MiB/sec (6407.65 MB/sec) </pre> </div></div> <p>I've observed that if you add two ports on the same HCA as different interfaces to the LNet network there is no performance boost. Performance boost is only seen when you add two different physical HCA cards. Not 100% sure why that is.</p> <p>A read test would do an RDMA write from the server to the client. Have you tried a write selftest from the two servers to the client? I'm wondering if you'd get the 11GB/s performance in this case.</p> <p><font color="#1f497d">Perhaps this is why two ports on one HCA are not scaling?</font></p> <p><font color="#1f497d">Examine the </font><font color="#1f497d">card slot:</font></p> <p><font color="#1f497d">One PCIe gen3 lane has max electrical signaling bandwidth of 984.6 MB/s.</font><font color="#1f497d">One “PCIe gen3 x16” slot has sixteen lanes: 16 * 984.6 = 15.75 GB/s max (guaranteed not to exceed)</font></p> <p><font color="#1f497d">And the dual-port HCA:<br/> </font><font color="#1f497d">One dual-port EDR-IB card requires a x16 slot but “offers” two 100Gb/s (12.5 GB/s) ports.</font><font color="#1f497d">Data encoding allows 64 of 66 bits to be used; 2 bits are for error correction.</font><font color="#1f497d">12.5 GB/s max * (64/66) leaves 12.1 GB/s usable bandwidth for one port to run at full speed.<br/> </font></p> <p><font color="#1f497d">Therefore, the 15.75 GB/s “x16” slot only allows one port to run at full 12.1 GB/s EDR-IB speed. </font><font color="#1f497d">Cabling both ports, and assigning LNETs to both ports, without LNET knowing how to apportion bandwidth among the two ports, seems problematic. </font><font color="#1f497d">The x16 slot only provides ~65% of the bandwidth required to run both ports at speed.</font></p> <p>Actually, I understood that there is PCI bandwdith limiation on dual port HCA, but 3-4GB is REALLY lower than expected single EDR bandwdith and I was suspecting something NUMA or NUMA/IO or CPT related problem behind. I don't want to get higher bandwdith by number of HCA here, but I am trying number of configurations (e.g. increasing peers, pin CPT to interface, use an dedicate CPT, etc) since as I said before, we are getting better performance on single EPYC client, but once added another CPU, read performance drops.</p> Related LU-12194 Development Rank 1|i00l9b: Rank (Obsolete) 9223372036854775807 Severity