https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9885 Lustre <p>After metadata performance improvements on <a href="https://jira.whamcloud.com/browse/LU-9796" title="Speedup file creation under heavy concurrency " class="issue-link" data-issue-key="LU-9796"><del>LU-9796</del></a>, we are hitting Lustre level bottleneck at File creation for scale.<br/> collected cost of all Lustre functions at file creation, ptlrpc_wait_event() took huge a mount of time cost.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@mds05 ~]# /work/ihara/perf-tools/bin/funccost -L Start setting lustre functions... Setting lustre functions done... Tracing "[lustre modules functions]"... Ctrl-C to end. /work/ihara/perf-tools/bin/funccost: line 158: tracing_enabled: Permission denied ^C FUNC TOTAL_TIME(us) COUNT AVG(us) ptlrpc_wait_event 1299908760 1745217 744.84 ptlrpc_server_handle_request 279979924 876419 319.46 tgt_request_handle 268911684 637004 422.15 tgt_enqueue 230880245 318456 725.00 ldlm_lock_enqueue 230872744 1273842 181.24 ldlm_handle_enqueue0 230791830 318456 724.72 mdt_intent_policy 224904050 318456 706.23 mdt_intent_opc 223658324 318456 702.32 mdt_intent_reint 223196495 318456 700.87 </pre> </div></div> <p>Why is ptlrpc_wait_event() taking a ot costs here?</p> LU-9885

Huge amount of costs in ptlrpc_wait_event() at file creation

Bug Minor Open Unresolved Peter Jones Shuichi Ihara Wed, 16 Aug 2017 04:14:33 +0000 Wed, 1 Sep 2021 17:25:29 +0000 Lustre 2.7.0 0 8 <p>Is this CPU time or elapsed wall clock time? The client threads are all sleeping in ptlrpc_wait_event() for the reply from the MDS. Having more RPCs in flight means that more threads will be waiting in parallel, even if it increases the aggregate create rate. </p> <p>If this is elapsed wall-clock time, then that is normal and not much can be done about it. If this is CPU time, then it might be made more efficient. </p> <p>Hi Andreas,</p> <p> We used ftrace function profile which should not include sleep time, profile results<br/> here should be CPU time.</p> <p>Is this on 2.7.0 or a descendant? Could you describe the exact lustre revision and test used here?</p> <p>Do you see the same behaviour with Lustre 2.10?</p> <p>I was looking into the <tt>ptlrpc_wait_event()</tt> code, and the only part that can take a lot of CPU time is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> l_wait_event_exclusive_head(svcpt-&gt;scp_waitq, ptlrpc_thread_stopping(thread) || ptlrpc_server_request_incoming(svcpt) || ptlrpc_server_request_pending(svcpt, <span class="code-keyword">false</span>) || ptlrpc_rqbd_pending(svcpt) || ptlrpc_at_check(svcpt), &amp;lwi); </pre> </div></div> <p>and of all those checks, the only one that is not just a simple variable/flag check is <tt>ptlrpc_server_request_pending()</tt>. It looks like this may be hooking into the NRS code. Is it possible you have some NRS TBF patches/policies on this system that are causing this function to be slow? Alternately, it may be that if there is memory pressure on the MDS it may be that the <tt>ptlrpc_wait_event()</tt> is just being called a lot due to the 0.1s timeout being set in this case:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ptlrpc_rqbd_pending(svcpt) &amp;&amp; ptlrpc_server_post_idle_rqbds(svcpt) &lt; 0) { /* I just failed to repost request buffers. * Wait <span class="code-keyword">for</span> a timeout (unless something <span class="code-keyword">else</span> * happens) before I <span class="code-keyword">try</span> again */ svcpt-&gt;scp_rqbd_timeout = cfs_time_seconds(1) / 10; CDEBUG(D_RPCTRACE, <span class="code-quote">"Posted buffers: %d\n"</span>, svcpt-&gt;scp_nrqbds_posted); } </pre> </div></div> <p>What would be useful here is a <a href="http://www.brendangregg.com/flamegraphs.html" class="external-link" target="_blank" rel="nofollow noopener">flame graph</a> which shows where the CPU time is actually being spent.[</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>http://www.brendangregg.com/flamegraphs.html%20flame%20graph]</td> </tr> </tbody></table> </div> <p>For what it's worth, I often see a lot of time spent here when dealing with high rates of small messages. In my experience, when looking more closely, the culprit is the time spent doing the sleep/wake. My data is client side, but the code is the same.</p> <p>I'm not familiar with the tool you're using, does it break down the time spent in that function further (especially in what that function calls)? If not, I'd suggest using perf record/perf report (the flame graphs Andreas mentioned are a way to visualize the sort of data it generates) - it makes it easy to see where time is spent in a function and functions called from it. If you are just charting time spent, then if the time is spread out between a few sub calls, they could be far down the list but together account for the time in that function.</p> <p>Thanks Andreas and Patrick<br/> Yes, we just used a simple tool perf-tool to confirm whether if there are any places where are a lot of CPU consuming. Agreed, for now, we need to break down of each function for each CPU time. will try perftool to find what exact places for it.</p> Related LU-14976 Development Rank 1|hzzik7: Rank (Obsolete) 9223372036854775807 Severity