https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13287 Lustre <p>While testing in an environment with a single parent directory following by 1 shared sub directory for all client mdtest ranks, we are observe very little scaling when moving to more than 2 MDTs. See below for 1 million objects per MDT, 0K File Creates:</p> <p>1 MDTs - 83,948<br/> 2 MDTs - 115,929<br/> 3 MDTs - 123,186<br/> 4 MDTs - 130,846</p> <p>Stats and deletes are showcasing similar results. It seems to not follow a linear scale but instead plateaus. It would also seem that we are not the only ones to observe this. A recent Cambridge University IO-500 presentation presented a slide with very similar results (fourth from the bottom): <a href="https://www.eofs.eu/_media/events/lad19/03_matt_raso-barnett-io500-cambridge.pdf" class="external-link" target="_blank" rel="nofollow noopener">https://www.eofs.eu/_media/events/lad19/03_matt_raso-barnett-io500-cambridge.pdf</a></p> LU-13287

DNE2 - Shared directory performance does not scale and starts to plateau beyond 2MDTs

Improvement Major Resolved Not a Bug WC Triage Petros Koutoupis dne2 llnl performance Fri, 21 Feb 2020 20:39:08 +0000 Wed, 20 May 2020 00:00:14 +0000 Wed, 20 May 2020 00:00:06 +0000 Lustre 2.13.0 0 7 <p>Possibly also reported and related to <a href="https://jira.whamcloud.com/browse/LU-9436" title="DNE2 - performance improvement with wide stripping directory" class="issue-link" data-issue-key="LU-9436">LU-9436</a>.</p> <p>Is this with one MDT per MDS, or are all four MDTs on the same MDS? If all MDTs are on the same MDS, then this is totally expected, as there just isn't enough unused CPU/network on the MDS to double or quadruple the performance on that node.</p> <p>How many clients are being used for this test? Does the performance improve when there are additional clients added for the 3/4 MDT test cases? Having the actual test command line included in the problem description would make this report a lot more useful.</p> <p>Looking at the referenced slide from the Cambridge presentation (attached), it actually shows almost linear scaling for additional MDTs (one per MDS) up to 48, excluding the 48-MDT stat test. I suspect in that case they didn't have enough clients to drive the aggregate MDT performance to saturation.</p> <p><span class="image-wrap" style=""><a id="34327_thumb" href="https://jira.whamcloud.com/secure/attachment/34327/34327_Screen+Shot+2020-02-21+at+16.11.52.png" title="Screen Shot 2020-02-21 at 16.11.52.png" file-preview-type="image" file-preview-id="34327" file-preview-title="Screen Shot 2020-02-21 at 16.11.52.png"><img src="https://jira.whamcloud.com/secure/thumbnail/34327/_thumb_34327.png" style="border: 0px solid black" role="presentation"/></a></span> </p> <p>Andreas,</p> <p> </p> <p>&gt; Is this with one MDT per MDS, or are all four MDTs on the same MDS? </p> <p>1 MDT per MDS (each on one).</p> <p> </p> <p>&gt; How many clients are being used for this test?</p> <p>It was 60 clients.</p> <p> </p> <p>&gt; Does the performance improve when there are additional clients added for the 3/4 MDT test cases? </p> <p>We have not added more clients than this.</p> <p> </p> <p>&gt; Having the actual test command line included in the problem description would make this report a lot more useful. </p> <p>We had four MDTs</p> <p>lfs mkdir -c 4 &lt;remote directory&gt; <span class="error">&#91;-D&#93;</span></p> <p>mdtest -i 3 -p 30 -F -C -E -T -r -n $(( 1048576 / $PROCS )*Num_MDTs) -v -d $&lt;remote directory/OUTDIR&gt;</p> <p> </p> <p>We will do 1 million objects per MDT…so for this test, we have 4 MDTs, so we did 4 Million objects. Again, 60 Clients.</p> <p>With the same mdtest with -u flag, we see good scaling with 4 MDTs, remove the -u flag to not do unique directory operation per rank (shared directory), the lack of scaling is present. We even tried mdtest with and without -g flag <span class="error">&#91;in the mainline latest builds&#93;</span>, same behavior.</p> <p> </p> <p>&gt; Looking at the referenced slide from the Cambridge presentation (attached), it actually shows almost linear scaling for additional MDTs (one per MDS) up to 48, excluding the 48-MDT stat test. </p> <p>The scaling in the presentation is very minimal as it was in some of our older tests with larger MDT/client counts (up to 512 clients). Is this to be expected?</p> <p>&gt;&gt; Andreas wrote:</p> <blockquote><p>Looking at the referenced slide from the Cambridge presentation (attached), it actually shows almost linear scaling for additional MDTs (one per MDS) up to 48, excluding the 48-MDT stat test. </p></blockquote> <p>&gt; Petros wrote:</p> <blockquote><p>The scaling in the presentation is very minimal as it was in some of our older tests with larger MDT/client counts (up to 512 clients). Is this to be expected?</p></blockquote> <p>More specifically, the scaling in the chart is about the easy mdtest (shared dir?) and stats. I think the focus of the problem is the scaling/performance of create in a single shared directory.</p> <p>We haven't tested this with recent 2.12 or master, but we also saw cases of poor DNE2 scaling in the past.</p> <p>@Olaf Faaland,</p> <p>I have tested master from 3 weeks ago or so and the results are the same. Part of the challenges that I am facing here is: how much of this minimal scaling is expected and how much room do we have to make it better? Earlier presentations posted online show that between 1-4 MDTs running mknod tests show some scaling but these were running an older build of Lustre and since then our single MDT performance has gotten exponentially better. Today, when I run mknod tests, the scaling results are no different than my creates.</p> <p>I have attached some flamegraphs and perf reports (MDT-DNE2-shareddir_flamegraphs.zip) for a 2 MDT and then 4 MDT configuration during load. Anyway, I am able to provide any data or traces upon request. Also, I have run the same tests using normal creates and then again with mknod with the same scaling results. Any thoughts, ideas, etc?</p> <p>I also shared shared-directory_mdt-perf.tar.gz, which consists of the flamegraphs of the original test that correlate to the numbers posted above in the description. Note that in the tarball, mdt0-1total consists of the single MDT testing while the rest of the subdirectories inside the archive are each MDT in a 4 MDT configuration.</p> <p>Added the tarball archive_smaller_inodes-tests.tar.gz and an accompanying powerpoint archive_smaller_inodes-tests.pptx which highlights DNE2 single shared directory scaling utilizing the large Moon cluster over at LANL. We were able to drive load from 512 clients and starting from a single server, double it at each iteration until we reached 32 MDTs. With enough clients, it seems that there was a reasonable amount of scaling and that this issue becomes much less of a concern. I will close this ticket unless there are objections to my doing so.</p> <p>@Andreas Dilger,</p> <p>It seems that I do not have the proper rights to close this ticket. Please advise.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=koutoupis" class="user-hover" rel="koutoupis">koutoupis</a> try again</p> <p>@Peter Jones</p> <p>Still cannot close the ticket. It is not even an option.</p> <p>Sorry about that - two similarly named groups lured me into an error. Please have another go - I think I got it this time</p> <p>It would seem that with enough client load, that we are able to drive proper DNE2 single shared directory scaling.</p> <p>Reopen to change resolution.</p> Related LU-9436 Development Rank 1|i00u8n: Rank (Obsolete) 9223372036854775807