https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11164 Lustre <p>The lvbo methods have to reallocate <tt>lu_env</tt> every time, which can be quite expensive in terms of CPU cycles at scale.<br/> The layers above can pass <tt>lu_env</tt> to reuse existing one.</p> LU-11164

lvbo_*() methods to reuse env

Improvement Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev perf_optimization Mon, 23 Jul 2018 08:02:20 +0000 Fri, 1 Mar 2019 16:01:18 +0000 Sat, 6 Oct 2018 13:23:47 +0000 Lustre 2.12.0 0 9 <p>Patch is at <a href="https://review.whamcloud.com/#/c/32832/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32832/</a></p> <p>I see this is marked as upstream. Is that correct?</p> <p>No it's not - thanks for point that out!</p> <p> </p> <p>Patrick,<br/> attached is a flame graph ( <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31095/31095_ost_io.svg" title="ost_io.svg attached to LU-11164">ost_io.svg^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> ) showing CPU usage for the OST under a high-throughput random read workload (fake IO used so that no storage overhead is present, just network and RPC processing). In <tt>ost_lvbo_update()</tt> the <tt>lu_env_init()</tt> and <tt>lu_env_fini()</tt> functions are consuming over 10% of the OSS CPU for basically no benefit. The <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31096/31096_master-patch-32832.svg" title="master-patch-32832.svg attached to LU-11164">master-patch-32832.svg^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> flame graph shows the <tt>ost_lvbo_update()</tt> CPU usage is down to 1.5% when the patch is applied, which resulted in a 6.3% performance improvement for random 4KB reads. Alex, could you please include these results in the commit comment so that it is more clear why we want to land that patch.</p> <p>The <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31095/31095_ost_io.svg" title="ost_io.svg attached to LU-11164">ost_io.svg^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> graph is also showing <tt>find_or_create_page()</tt> using 4.25% of CPU, which drove the creation of patch <a href="https://review.whamcloud.com/32875" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32875</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11347" title="Do not use pagecache for SSD I/O when read/write cache are disabled" class="issue-link" data-issue-key="LU-11347"><del>LU-11347</del></a> osd: do not use pagecache for I/O</tt>".</p> <p>Here is performance resutls with patches.<br/> 4K ranodm read from 768 processes on 32 clients with fio below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[randread] ioengine=sync rw=randread blocksize=4096 iodepth=16 direct=1 size=4g runtime=60 numjobs=24 group_reporting directory=/cache1/fio.out filename_format=f.$jobnum.$filenum </pre> </div></div> <p>master without patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>All clients: (groupid=0, jobs=32): err= 0: pid=0: Mon Oct 1 09:00:53 2018 read: IOPS=564k, BW=2205Mi (2312M)(129GiB/60006msec) clat (usec): min=210, max=228722, avg=1370.30, stdev=623.75 lat (usec): min=210, max=228722, avg=1370.45, stdev=623.75 bw ( KiB/s): min= 1192, max=11160, per=0.13%, avg=2911.36, stdev=1003.58, samples=91693 iops : min= 298, max= 2790, avg=727.83, stdev=250.90, samples=91693 lat (usec) : 250=0.01%, 500=3.09%, 750=4.03%, 1000=11.72% lat (msec) : 2=71.92%, 4=9.13%, 10=0.06%, 20=0.05%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=0.19%, sys=1.70%, ctx=33884824, majf=0, minf=49349 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% issued rwts: total=33867532,0,0,0 short=0,0,0,0 dropped=0,0,0,0 </pre> </div></div> <p>master with patch 32832</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>All clients: (groupid=0, jobs=32): err= 0: pid=0: Mon Oct 1 00:50:54 2018 read: IOPS=639k, BW=2497Mi (2619M)(146GiB/60062msec) clat (usec): min=202, max=303137, avg=1199.86, stdev=731.66 lat (usec): min=202, max=303137, avg=1200.01, stdev=731.65 bw ( KiB/s): min= 624, max=12712, per=0.13%, avg=3319.10, stdev=1131.37, samples=91719 iops : min= 156, max= 3178, avg=829.76, stdev=282.84, samples=91719 lat (usec) : 250=0.01%, 500=4.35%, 750=4.64%, 1000=21.49% lat (msec) : 2=65.65%, 4=3.70%, 10=0.07%, 20=0.08%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=0.21%, sys=1.89%, ctx=38418553, majf=0, minf=54483 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% issued rwts: total=38398687,0,0,0 short=0,0,0,0 dropped=0,0,0,0 </pre> </div></div> <p>564k vs 639k IOPS. patch contributes 13% performance gain.</p> <p>Thanks for the detailed benchmark info, Andreas, Ihara.  (Sorry, I was not originally following the LU so did not see the earlier comment.)</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32832/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32832/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11164" title="lvbo_*() methods to reuse env" class="issue-link" data-issue-key="LU-11164"><del>LU-11164</del></a> ldlm: pass env to lvbo methods<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e02cb40761ff8aae3df76c4210a345420b6d4ba1</p> <p>Before we close this has anyone measured metadata performance? I wonder if their was any impact form this work.</p> <p>Landed for 2.12. James, perf testing is a standard part of release testing</p> <p>I'm seeing significant performance drop (-15%) in createmany -o between 2.11 and 2.12. Testing historical commits with a small step shows about 5% performance drop right after this patch.</p> <p>any profiling information?</p> <p>There's some profiling information collected during testing in automatic mode. I hope it can be useful.</p> <p>There's no significant difference in open(O_CREAT) processing time, but close() seems to have become slower:</p> <p><a href="https://jira.whamcloud.com/browse/LU-11164" title="lvbo_*() methods to reuse env" class="issue-link" data-issue-key="LU-11164"><del>LU-11164</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - open/close 5948 (time 1551093589.39 total 10.00 last 594.79) - open/close 10000 (time 1551093596.20 total 16.82 last 594.22) - open/close 15658 (time 1551093606.20 total 26.82 last 565.79) total: 20000 open/close in 34.55 seconds: 578.94 ops/second ll_atomic_open:22765503.0:20000:1138.0 ll_file_release:9271202.0:20000:463.0 ll_inode_revalidate:850446.0:40005:21.0 ll_file_open:326223.0:20000:16.0 lmv_intent_lock:21263116.0:60005:354.0 mdc_enqueue_base:20668025.0:20002:1033.0 mdc_close:8846957.0:20000:442.0 mdc_intent_lock:21006833.0:60005:350.0 ldlm_cli_enqueue:19807126.0:20002:990.0 ldlm_handle_enqueue0:12404488.0:20002:620.0 ldlm_cli_enqueue_fini:870123.0:20002:43.0 ptlrpc_queue_wait:27100485.0:40004:677.0 ptlrpc_server_handle_request:19790070.0:66468:297.0 mdt_reint_open:11116636.0:20000:555.0 mdt_close:2967136.0:20000:148.0 mdt_intent_getattr:147.0:2:73.0 mdd_create:5972754.0:20000:298.0 mdd_create_object:2055115.0:20000:102.0 mdd_trans_start:1641342.0:60000:27.0 mdd_trans_stop:1128474.0:60000:18.0 mdd_update_time:131643.0:20000:6.0 mdd_changelog_ns_store:11204.0:20000:0.0 ldiskfs_create_inode:1199474.0:20000:59.0 </pre> </div></div> <p>pre-<a href="https://jira.whamcloud.com/browse/LU-11164" title="lvbo_*() methods to reuse env" class="issue-link" data-issue-key="LU-11164"><del>LU-11164</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - open/close 6245 (time 1551094193.65 total 10.00 last 624.45) - open/close 10000 (time 1551094199.67 total 16.02 last 623.75) - open/close 15804 (time 1551094209.68 total 26.02 last 580.35) total: 20000 open/close in 33.23 seconds: 601.83 ops/second ll_atomic_open:22203052.0:20000:1110.0 ll_file_release:8576370.0:20000:428.0 ll_inode_revalidate:823214.0:40005:20.0 ll_file_open:321143.0:20000:16.0 lmv_intent_lock:20684255.0:60005:344.0 mdc_enqueue_base:20090571.0:20002:1004.0 mdc_close:8149427.0:20000:407.0 mdc_intent_lock:20431136.0:60005:340.0 ldlm_cli_enqueue:19175889.0:20002:958.0 ldlm_handle_enqueue0:12598163.0:20002:629.0 ldlm_cli_enqueue_fini:908190.0:20002:45.0 ptlrpc_queue_wait:25720604.0:40004:642.0 ptlrpc_server_handle_request:19790636.0:41798:473.0 mdt_reint_open:11314481.0:20000:565.0 mdt_close:2930872.0:20000:146.0 mdt_intent_getattr:157.0:2:78.0 mdd_create:6212753.0:20000:310.0 mdd_create_object:2052026.0:20000:102.0 mdd_trans_start:1929037.0:60000:32.0 mdd_trans_stop:1121551.0:60000:18.0 mdd_update_time:130001.0:20000:6.0 mdd_changelog_ns_store:9368.0:20000:0.0 ldiskfs_create_inode:1182995.0:20000:59.0 </pre> </div></div> <p>open times look similar (1138.0 us vs 1110.0 us per operation, total 22.7 mln us vs 22.2 mln us), but close times have changed (463 us vs 428 us, 10% difference, total 9.2 mln us vs 8.5 mln us).</p> <p>The results stay similar for the neighbor commits.</p> <p>did you try to collect some data with say oprofile?</p> <p>yet another option would be to collect rpc_stats on the server side to see is it on the server or client side.</p> <blockquote><p>did you try to collect some data with say oprofile?</p></blockquote> <p>No, why? It's an outdated tool. The ftrace profiling data pasted above is more accurate.</p> <p>I will collect additional data for this case if you have no idea why performance dropped.</p> <p>at the moment I don't understand why passing extra argument to few methods would affect MDS performance that much.<br/> will keep investigating..<br/> any additional data is very appreciated.</p> <p>Still create ost objects unless there is -m (mknod) option? let me confirm with/without patch.</p> <blockquote><p>Still create ost objects unless there is -m (mknod) option? let me confirm with/without patch.</p></blockquote> <p>The directory in the test is configured for the DoM layout:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lfs setstripe -E 1048576 -L mdt . # createmany -o file 1 20000 </pre> </div></div> <p>I'm not an expert in this patch, but isn't it performing lu_env_init()/lu_env_fini() in ptlrpc_set_wait() when this sequence isn't really needed?</p> <p>This observation seems to be backed by the posted profiling data:</p> <p>mdt_close() execution times are equal with and without the patch: 146/148 us, the differences between ll_file_release() and mdc_close() execution times are equal too: 463-442=21=428-407.</p> <p>Apparently, the observed performance drop happens between mdc_close() and mdt_close() which smoothly correlates with the possible lu_env_init()/lu_env_fini() overhead in ptlrpc_set_wait().</p> <p>The following change restores createmany -o performance to that of pre <a href="https://jira.whamcloud.com/browse/LU-11164" title="lvbo_*() methods to reuse env" class="issue-link" data-issue-key="LU-11164"><del>LU-11164</del></a> for me:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/ptlrpc/client.c b/lustre/ptlrpc/client.c index 1ea3b9fdc1..f2a994c707 100644 --- a/lustre/ptlrpc/client.c +++ b/lustre/ptlrpc/client.c @@ -2327,7 +2327,9 @@ <span class="code-object">int</span> ptlrpc_set_wait(<span class="code-keyword">const</span> struct lu_env *env, struct ptlrpc_request_set *set) struct list_head *tmp; struct ptlrpc_request *req; struct l_wait_info lwi; +#<span class="code-keyword">if</span> 0 struct lu_env _env; +#endif time64_t timeout; <span class="code-object">int</span> rc; ENTRY; @@ -2345,6 +2347,7 @@ <span class="code-object">int</span> ptlrpc_set_wait(<span class="code-keyword">const</span> struct lu_env *env, struct ptlrpc_request_set *set) <span class="code-keyword">if</span> (list_empty(&amp;set-&gt;set_requests)) RETURN(0); +#<span class="code-keyword">if</span> 0 /* ideally we want env provide by the caller all the time, * but at the moment that would mean a massive change in * LDLM <span class="code-keyword">while</span> benefits would be close to zero, so just @@ -2356,6 +2359,7 @@ <span class="code-object">int</span> ptlrpc_set_wait(<span class="code-keyword">const</span> struct lu_env *env, struct ptlrpc_request_set *set) RETURN(rc); env = &amp;_env; } +#endif <span class="code-keyword">do</span> { timeout = ptlrpc_set_next_timeout(set); @@ -2436,8 +2440,10 @@ <span class="code-object">int</span> ptlrpc_set_wait(<span class="code-keyword">const</span> struct lu_env *env, struct ptlrpc_request_set *set) rc = req-&gt;rq_status; } +#<span class="code-keyword">if</span> 0 <span class="code-keyword">if</span> (env &amp;&amp; env == &amp;_env) lu_env_fini(&amp;_env); +#endif RETURN(rc); } </pre> </div></div> <p>Alex, Shuichi, what do you think about removing this allocation? Is it crucial for this patch?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=panda" class="user-hover" rel="panda">panda</a> , I'm testing <a href="https://review.whamcloud.com/#/c/34333/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34333/</a></p> <p>Given that <a href="https://jira.whamcloud.com/browse/LU-11164" title="lvbo_*() methods to reuse env" class="issue-link" data-issue-key="LU-11164"><del>LU-11164</del></a> landed in 2.12.0. which is now released it would be very helpful to move this discussion to a new Jira ticket linked to this one. That will make it easier to track getting the fix landed and included in a 2.12.x maintenance release - thanks!</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=panda" class="user-hover" rel="panda">panda</a> would you please create a new ticket like Peter suggested?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a>, done: <a href="https://jira.whamcloud.com/browse/LU-12034" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-12034</a></p> <p>Thanks Andrew!</p> Related LU-11347 LU-12034 Development Rank 1|hzzzjz: Rank (Obsolete) 9223372036854775807