https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6580 Lustre <p>Hi,</p> <p>With Lustre 2.7.0, we have noticed a very bad impact of the number of ptlrpcd threads on the client over the data read performance.<br/> If the number of ptlrpcd threads on the client is set to its default value which is the number of cores, then we have a drop in performance of almost 20%, compared to the case where we reduce the number of ptlrpcd threads to 1/3 of the number of cores.</p> <p>For instance see the results attached to this ticket. We use IOR with 24 tasks on a Lustre client node with 24 cores and one Infiniband FDR interface.</p> <p>What can explain this phenomenon in Lustre 2.7.0, as we do not see it with Lustre 2.5 and 2.6?<br/> Do you consider this is a bug, or a behavior change due to some modifications in Lustre 2.7 code (in that case we would like to learn more about)?</p> <p>Thanks in advance,<br/> Sebastien.</p> LU-6580

Poor read performance with many ptlrpcd threads on the client

Bug Major Resolved Duplicate Dmitry Eremin Sebastien Buisson Thu, 7 May 2015 14:57:31 +0000 Wed, 13 Oct 2021 03:07:18 +0000 Wed, 13 Oct 2021 03:07:18 +0000 Lustre 2.7.0 0 9 <p>Can you please take a look at the patch in <a href="https://jira.whamcloud.com/browse/LU-6325" title="CPT bound ptlrpcd&#39;s are unimplemented" class="issue-link" data-issue-key="LU-6325"><del>LU-6325</del></a>, which improves the NUMA affinity of the ptlrpcd threads?</p> <p>That doesn't explain why there is a regression in 2.7.0, which still needs to be looked into, but may help fix the problem. </p> <p>Hi,</p> <p>I gave a try to patch from <a href="https://jira.whamcloud.com/browse/LU-6325" title="CPT bound ptlrpcd&#39;s are unimplemented" class="issue-link" data-issue-key="LU-6325"><del>LU-6325</del></a> at <a href="http://review.whamcloud.com/13972" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13972</a>.</p> <p>I can see two interesting phenomena:</p> <ul> <li>when not restricting ptlrpcd threads to specific cpts, the write performance is good and stable, whatever the number of ptlrpcd threads. But the read performance is very low, lower than without the patch. See graph ior_24tasks_lu6325_allcpt.png.</li> <li>when restricting ptlrpcd threads to the cpt local to the network adapter, the write performance slightly decreases, but the read performance is much better, and does not drop when I increase the number of ptlrpcd threads. See ior_24tasks_lu6325_cpt0.png.</li> </ul> <p>These results are obtained with 'ptlrpcd_partner_group_size=1', because using the default value (2) gives slightly lower performance.</p> <p>So the ability to restrict ptlrpcd threads to a specific cpt (thanks to patch <a href="http://review.whamcloud.com/13972" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13972</a>) is very helpful. But this brings a new question: why read performance is so poor when putting ptlrpcd threads on all cpts?</p> <p>Thanks,<br/> Sebastien.</p> Related LU-6325 Development End date Fri, 17 Jul 2015 14:57:31 +0000 Rank 1|hzxcqv: Rank (Obsolete) 9223372036854775807 Severity Start date Thu, 7 May 2015 14:57:31 +0000