https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1408 Lustre <p>This is a single client performance regression on 2.2 compared to 2.1.2 or 1.8.x.</p> <p>I filed <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a> before for another single client's performance regression, but that regression also happened on 2.1.x as well as 2.2 when the amount of file size is larger than client's memory size.</p> <p>So, this regression might not be related to <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>, but there is an another regression on 2.2 even if amount of file size is smaller than client's memory size.</p> lustre-2.2, b2_1 and lustre-1.8.7<br/> CentOS6.2 on both servers and clients LU-1408

single client's performance regression test

Bug Blocker Resolved Fixed Hongchao Zhang Shuichi Ihara Tue, 15 May 2012 09:41:29 +0000 Mon, 2 Jun 2014 15:02:01 +0000 Fri, 20 Jul 2012 08:53:40 +0000 Lustre 2.3.0 Lustre 2.6.0 0 9 <p>test script with IOR</p> <p>Here is test scripts and an initial single client performance testing on various lustre version.<br/> <a href="http://jira.whamcloud.com/secure/attachment/11360/test-script.sh" class="external-link" rel="nofollow">http://jira.whamcloud.com/secure/attachment/11360/test-script.sh</a><br/> <a href="http://jira.whamcloud.com/secure/attachment/11303/lustre-singleclient-comparison.xlsx" class="external-link" rel="nofollow">http://jira.whamcloud.com/secure/attachment/11303/lustre-singleclient-comparison.xlsx</a></p> <p>Servers are running with lustre-2.2 on CentOS6.2, just test each checksum algo with various lustre version.</p> <p>During test for <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a>, I saw another big performance differences between b2_1 branch and 2.1.2RC0 tag on the single client. Please refer to <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a>.</p> <p>Oleg</p> <p>Could you please suggest some steps here and perhaps someone else can assist in executing them?</p> <p>Thanks</p> <p>Peter</p> <p>This regression is related to <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a>.</p> <p>As I commented on <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a>, we also saw the single client's performance regression on the latest b2_1 branch, but no regression on 2.1.2RC0 branch. The regression started from commit b9cbe3616b6e0b44c7835b1aec65befb85f848f9 (<a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> debug: reduce stack usage). Please see <a href="https://jira.whamcloud.com/browse/LU-1413" title="difference of single client&#39;s performance between b2_1 and 2.1.2RC0" class="issue-link" data-issue-key="LU-1413"><del>LU-1413</del></a> for these test results.</p> <p><a href="http://jira.whamcloud.com/secure/attachment/11303/lustre-singleclient-comparison.xlsx" class="external-link" rel="nofollow">http://jira.whamcloud.com/secure/attachment/11303/lustre-singleclient-comparison.xlsx</a><br/> Above, my testing on b2_2 and b2_1 and 1.8.7, I used 2.1.2RC0 branch for b2_1 testing.</p> <p>This is why 2.1.2 numbers were faster than b2_2, because <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> was not landed in 2.1.2RC0 yet, but landed in b2_2.</p> <p>In order to verify, I removed commit for <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> from b2_2 and measured the performance.<br/> (IOR with 4 processes with checksum=disable)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> WRITE(GB/s) READ(GB/s) b2_2 1.4 1.4 b2_2/wo LU969 2.4 2.8 b2_1 1.4 1.4 b2_1/wo LU969 2.7 3.2 </pre> </div></div> <p>I will try to run full testing on FDR infiniband.</p> <p>This is benchmark results on 2.1.2RC0, b2_2 and b2_2 without <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patches.<br/> RHEL6.2, Lustre-2.2 for servers. RHEL6.2 on an client (96GB memory, 12 CPU cores) with QDR Infiniband.</p> <p>All performance were improved by disabling <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patches.</p> <p>benchmark results with correct parameters.</p> <p><a href="http://jira.whamcloud.com/secure/attachment/11413/single-client-perforamnce-LU1408-rev2.xlsx" class="external-link" rel="nofollow">http://jira.whamcloud.com/secure/attachment/11413/single-client-perforamnce-LU1408-rev2.xlsx</a></p> <p>This is correct benchmark results with the correct parameters. max_rpcs_in_flight=256 was missing on the previous benchmark results when the checksum was enabled.<br/> As far as I can see the new results, when I removed <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patches from b2_2, all numbers were alomost close to 2.1.2RC0's numbers.</p> <p>I will run benchmark on FDR Infiniband to see the maximum single perforamnce on b2_2.</p> <p>Ihahara</p> <p>Thanks for this investigation work!</p> <p>Hongchao</p> <p>Could you please look into why the <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> changes are negatively affecting performance?</p> <p>Thanks</p> <p>Peter</p> <p>this could be related to the modification of __CHECK_STACK,</p> <p>Hi,Ihahara<br/> could you please help to check whether there are much more logs in server side when testing with patches of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a>? thanks very much!</p> <p>I suspect the root problem is because we are filling the debug structure every time now regardless of if it hits or not.</p> <p>So perhaps if we move the structure filling until after the check, all should be fine?</p> <p>Soemthing like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/libcfs/include/libcfs/libcfs_debug.h b/libcfs/include/libcfs/libcfs_debug.h index 8a366f9..72171ad 100644 --- a/libcfs/include/libcfs/libcfs_debug.h +++ b/libcfs/include/libcfs/libcfs_debug.h @@ -203,12 +203,13 @@ static inline int cfs_cdebug_show(unsigned int mask, unsigned int subsystem) #define __CDEBUG(cdls, mask, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, cdls); \ \ - CFS_CHECK_STACK(&amp;msgdata, mask, cdls); \ + if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, cdls); \ \ - if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) \ + CFS_CHECK_STACK(&amp;msgdata, mask, cdls); \ libcfs_debug_msg(&amp;msgdata, format, ## __VA_ARGS__); \ + } \ } while (0) #define CDEBUG(mask, format, ...) __CDEBUG(NULL, mask, format, ## __VA_ARGS__) diff --git a/lustre/include/cl_object.h b/lustre/include/cl_object.h index 47782be..ad1cf7a 100644 --- a/lustre/include/cl_object.h +++ b/lustre/include/cl_object.h @@ -1065,9 +1065,9 @@ struct cl_page_operations { */ #define CL_PAGE_DEBUG(mask, env, page, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ cl_page_print(env, &amp;msgdata, lu_cdebug_printer, page); \ CDEBUG(mask, format , ## __VA_ARGS__); \ } \ @@ -1078,9 +1078,9 @@ do { \ */ #define CL_PAGE_HEADER(mask, env, page, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ cl_page_header_print(env, &amp;msgdata, lu_cdebug_printer, page); \ CDEBUG(mask, format , ## __VA_ARGS__); \ } \ @@ -1789,9 +1789,9 @@ struct cl_lock_operations { #define CL_LOCK_DEBUG(mask, env, lock, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ cl_lock_print(env, &amp;msgdata, lu_cdebug_printer, lock); \ CDEBUG(mask, format , ## __VA_ARGS__); \ } \ diff --git a/lustre/include/lu_object.h b/lustre/include/lu_object.h index 0fd61fb..b97a249 100644 --- a/lustre/include/lu_object.h +++ b/lustre/include/lu_object.h @@ -763,9 +763,9 @@ int lu_cdebug_printer(const struct lu_env *env, */ #define LU_OBJECT_DEBUG(mask, env, object, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ lu_object_print(env, &amp;msgdata, lu_cdebug_printer, object);\ CDEBUG(mask, format , ## __VA_ARGS__); \ } \ @@ -776,9 +776,9 @@ do { \ */ #define LU_OBJECT_HEADER(mask, env, object, format, ...) \ do { \ - LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ \ if (cfs_cdebug_show(mask, DEBUG_SUBSYSTEM)) { \ + LIBCFS_DEBUG_MSG_DATA_DECL(msgdata, mask, NULL); \ lu_object_header_print(env, &amp;msgdata, lu_cdebug_printer,\ (object)-&gt;lo_header); \ lu_cdebug_printer(env, &amp;msgdata, "\n"); \ </pre> </div></div> <p>at Toro(1 client, 1 MDT, 6 OST), several tests of b2_2 and previous b2_1(without the patch of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a>), the read/write<br/> performance is almost the same, then the affect of <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patch could be related to your test environment,</p> <p>could you please help to test with the following patch (against <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patch)? Thanks in advance!</p> <p>diff --git a/libcfs/include/libcfs/linux/libcfs.h b/libcfs/include/libcfs/linux/libcfs.h<br/> index ce07e80..0dadd84 100644<br/> &#8212; a/libcfs/include/libcfs/linux/libcfs.h<br/> +++ b/libcfs/include/libcfs/linux/libcfs.h<br/> @@ -79,7 +79,8 @@</p> <p> #define __CHECK_STACK(msgdata, mask, cdls) \<br/> do { \</p> <ul class="alternate" type="square"> <li>if (unlikely(CDEBUG_STACK() &gt; libcfs_stack)) { \<br/> + if (unlikely(CDEBUG_STACK() &gt; 3 * THREAD_SIZE / 4 &amp;&amp; \<br/> + CDEBUG_STACK() &gt; libcfs_stack)) { \<br/> libcfs_stack = CDEBUG_STACK(); \<br/> (msgdata)-&gt;msg_mask = D_WARNING; \<br/> (msgdata)-&gt;msg_cdls = NULL; \</li> </ul> <p>Hi Ihahara,</p> <p>could you please help to test this patch (<a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a>) ATM, thanks!</p> <p>ORNL is also testing this patch at this time on our test beds. Will post results soon. Thanks for the patch.</p> <p>Hongchao, I'm very sorry, our test system is shutdowning down on this week, and I was looking for other system, but not luck.</p> <p>James, I appreciate your helsp for testing!</p> <p>James, <br/> For tesitng, there is my IOR command. It's basic, but the block size is needed to optimize by number of process for less than client's memory.<br/> mpirun -np 4 IOR -b Xg -t 1m -F -C -w -r -e -vv -o /lustre/ior.out/file</p> <p>Got it. So if the block size is greater than the client memory does this problem still exist. I did a earlier running with a block size much larger than the client memory. In that case I got results similar with and without the patch.</p> <p>Yes, in case of larger than client's memory size, that problem is still exist and it was filed on <a href="https://jira.whamcloud.com/browse/LU-744" title="Single client&#39;s performance degradation on 2.1" class="issue-link" data-issue-key="LU-744"><del>LU-744</del></a>.</p> <p>Tests run on build 6296, 106 clients <a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Kernel: 2.6.32-220.13.1.el6_lustre.g10a847d.x86_64 Lustre: jenkins-arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel-6296-gc1ba127-PRISTINE-2.6.32-220.13.1.el6_lustre.g10a847d.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 848 (8 per node) aggregate filesize = 848 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5992.72 5561.75 5837.06 195.23 148.93573 0 848 8 3 0 0 1 0 0 1 1073741824 1048576 910533066752 POSIX 0 read 5092.44 4625.07 4896.03 197.95 177.65520 0 848 8 3 0 0 1 0 0 1 1073741824 1048576 910533066752 POSIX 0 Finished: Thu May 24 11:40:39 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 848 (8 per node) aggregate filesize = 848 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 6858.42 6656.15 6766.77 83.66 128.34560 0 848 8 3 1 0 1 0 0 1 1073741824 1048576 910533066752 POSIX 0 read 6196.60 6069.53 6142.16 53.44 141.38652 0 848 8 3 1 0 1 0 0 1 1073741824 1048576 910533066752 POSIX 0 Finished: Thu May 24 11:58:08 2012 </pre> </div></div> <p>Tests run on build 6296, 50 clients: (includes patch)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel: 2.6.32-220.13.1.el6_lustre.g10a847d.x86_64 Lustre: jenkins-arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel-6296-gc1ba127-PRISTINE-2.6.32-220.13.1.el6_lustre.g10a847d.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 716.36 610.43 659.39 43.61 623.87034 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 5380.47 5285.04 5336.81 39.38 76.75416 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 14:16:16 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 6763.45 176.72 4552.72 3094.36 813.10982 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 5981.72 5336.92 5766.69 303.89 71.23411 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 15:02:52 2012 </pre> </div></div> <p>Tip of 2.1, includes patch, 50 clients: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel: 2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 Lustre: jenkins-g696f7f2-PRISTINE-2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5179.29 4777.87 4998.78 166.35 82.03194 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 5334.62 4381.00 4986.10 429.51 82.80132 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 19:36:41 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 6397.31 6191.68 6273.38 89.10 65.30478 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 6289.52 5801.98 5992.59 212.76 68.43547 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 19:44:19 2012 </pre> </div></div> <p>Tip of 2.1 (build #81)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel: 2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 Lustre: jenkins-g696f7f2-PRISTINE-2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 840 (8 per node) aggregate filesize = 840 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5770.81 5199.94 5412.42 254.87 159.26572 0 840 8 3 0 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 read 5151.68 5057.77 5117.10 42.14 168.10677 0 840 8 3 0 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 Finished: Thu May 24 20:06:38 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 840 (8 per node) aggregate filesize = 840 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 6177.47 5906.30 6009.84 119.62 143.18126 0 840 8 3 1 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 read 6144.79 5520.26 5738.26 287.71 150.26389 0 840 8 3 1 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 Finished: Thu May 24 20:23:21 2012 </pre> </div></div> <p><a href="http://review.whamcloud.com/2901" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2901</a> - build without the offending patch 50 clients</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel: 2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 Lustre: jenkins-ga944961-PRISTINE-2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5489.07 4984.54 5268.71 210.86 77.86901 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 5410.81 5175.37 5257.66 108.40 77.93804 0 400 8 3 0 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 22:09:52 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 400 (8 per node) aggregate filesize = 400 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 6250.72 3615.96 5349.70 1226.26 81.68548 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 read 6329.04 5786.43 6014.79 229.69 68.19637 0 400 8 3 1 0 1 0 0 1 1073741824 1048576 429496729600 POSIX 0 Finished: Thu May 24 22:18:16 2012 </pre> </div></div> <p>105 clients</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel: 2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 Lustre: jenkins-ga944961-PRISTINE-2.6.32-220.17.1.el6_lustre.g636ddbf.x86_64 parallel-scale.test_iorssf.test_log.hyperion244.log clients = 840 (8 per node) aggregate filesize = 840 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5372.92 5306.97 5340.69 26.95 161.06181 0 840 8 3 0 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 read 5136.44 5110.41 5121.05 11.14 167.96625 0 840 8 3 0 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 Finished: Thu May 24 22:39:19 2012 parallel-scale.test_iorfpp.test_log.hyperion244.log clients = 840 (8 per node) aggregate filesize = 840 GiB blocksize = 1 GiB xfersize = 1 MiB Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 5720.72 5560.95 5632.40 66.31 152.73755 0 840 8 3 1 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 read 6225.76 5433.65 5778.62 331.35 149.33184 0 840 8 3 1 0 1 0 0 1 1073741824 1048576 901943132160 POSIX 0 Finished: Thu May 24 22:56:19 2012 </pre> </div></div> <p>Please test on the single client with multiple thread, instead of multiple nodes.<br/> The original problem of this is single client's performance regression. So, even we could get better performance on the multiple clients, still need to make sure really single client's perforamnce regression is gone by patches. </p> <p>Oh I see. Just like Cliff I was not seeing really big difference between with and without <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> patch. Also like Cliff I was testing with multiple nodes of various thread counts per node. Thank for you for clarifying this. Will do another round of testing to see if this makes a difference.</p> <p>Finished a run against master (2.2.53). Cliff can you try running on a single client as well. The results I got were:</p> <p>*************************************************************************************<br/> No <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> patch<br/> *************************************************************************************</p> <p>16 Threads one single node<br/> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)<br/> --------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------<br/> write 607.24 568.29 596.03 14.05 151.81 142.07 149.01 3.51 110.01801<br/> read 723.09 700.82 714.84 8.14 180.77 175.20 178.71 2.04 91.69101</p> <p>Max Write: 607.24 MiB/sec (636.74 MB/sec)<br/> Max Read: 723.09 MiB/sec (758.21 MB/sec)</p> <p>*************************************************************************************<br/> With <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> patch<br/> *************************************************************************************<br/> 16 Threads one single node<br/> Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)<br/> --------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------<br/> write 611.51 445.17 569.68 62.68 152.88 111.29 142.42 15.67 116.72391<br/> read 771.25 746.29 757.22 9.74 192.81 186.57 189.30 2.44 86.56250</p> <p>Max Write: 611.51 MiB/sec (641.21 MB/sec)<br/> Max Read: 771.25 MiB/sec (808.71 MB/sec)</p> <p>hmm.. it seems to be pretty lower than I had even with patch. Server and clients are connected with QDR Infiniband? and how many CPU cores on the client? and disabled checksum just in case?</p> <p>hmm.. it seems to be pretty lower than I had even with patch. Server and clients are connected with QDR Infiniband? and how many CPU cores on the client? and disabled checksum just in case?</p> <p>Yes the fiber is QDR. Each node has 8 Intel Xeon CPU core (mode E5520) at 2.27GHz. Actually for this set of test I had checksums on. I just realized that. No matter; with checksum on or off the results are pretty close to each other.</p> <p>ok, our system is back and tested <a href="https://jira.whamcloud.com/browse/LU-1408" title="single client&#39;s performance regression test" class="issue-link" data-issue-key="LU-1408"><del>LU-1408</del></a> patches.<br/> Tested on an client (X5675, 3.07GHz, 48GB memory, QDR Infiniband), RHEL6.2, lustre-2.1.2-RC1 (both servers and client). </p> <p>Confirmed the patches fix the performance regression problem. Here is test results.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>===== without patch (original 2.1.2-RC1) ===== # mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file ... snip ... Max Write: 1401.19 MiB/sec (1469.26 MB/sec) # mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file ... snip ... Max Read: 1510.69 MiB/sec (1584.07 MB/sec) </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>===== with LU-1408 patch ===== # mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file ... snip ... Max Write: 2578.54 MiB/sec (2703.80 MB/sec) # pdsh -a "sync; echo 3 &gt; /proc/sys/vm/drop_caches" # mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file ... snip ... Max Read: 2663.67 MiB/sec (2793.06 MB/sec) </pre> </div></div> <p>Thanks for confirming the results.</p> <p>Can you please tell me which patches did you tests? Just the one in gerrit?</p> <p>I did test patch set 2 on <a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a><br/> After my testing is done and verified, I wondered if I could set "verified" flag as one of manual tester, but I couldn't find how.<br/> Any advises how we can do that? </p> <p>Hi Ihara</p> <p>could you please help to test the patch set 3 on <a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a>, the patch set 2 disables<br/> the stack check, which is enabled for non x86-64 architecture, thanks!</p> <p>Hi Ihara,</p> <p>the patch is updated, and could you please help to test patch set 5 on <a href="http://review.whamcloud.com/#change,2889?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889?</a> Thanks</p> <p>Hi Ihara,</p> <p>Have you tested the patch set 5 at <a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a>, which is a little different from patch set 2, thanks!</p> <p>Hongchao, sorry for delay... I will test patch soon. keep you updates here once my testing is done.<br/> any advices I can add "manual test" flag on <a href="http://review.whamcloud.com/#change,2889" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2889</a>, after test is done and if result is OK?</p> <p>Hi Ihara, I have add you as one of the reviewers of the patch, you can add review feedback according to the test result, thanks!</p> <p>Hi Hongchao,</p> <p>Just tested the patch with 2.1.2-RC1. (since <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> patches were droped on 2.1.2-RC2), the latest patch seems to be OK.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2.1.2-RC0 without any patches # mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file Max Write: 2537.56 MiB/sec (2660.83 MB/sec) # pdsh -a "sync; echo 3 &gt; /proc/sys/vm/drop_caches" # mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -r -k -vv -o /lustre/file Max Read: 2848.48 MiB/sec (2986.85 MB/sec) </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2.1.2-RC1 + patches # mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file Max Write: 2468.03 MiB/sec (2587.92 MB/sec) # pdsh -a "sync; echo 3 &gt; /proc/sys/vm/drop_caches" # mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -r -k -vv -o /lustre/file Max Read: 2881.06 MiB/sec (3021.01 MB/sec) </pre> </div></div> <p>Landed for 2.3</p> Related LU-1413 LU-744 LU-969 Development Rank 1|hzv6kf: Rank (Obsolete) 4598 Severity