https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16691 Lustre <p>In some cases, ldiskfs block allocation can consume a large amount of CPU cycles handling block allocations and cause OST threads to become blocked:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crmd[16542]: notice: High CPU load detected: 261.019989 crmd[16542]: notice: High CPU load detected: 258.720001 crmd[16542]: notice: High CPU load detected: 265.029999 crmd[16542]: notice: High CPU load detected: 270.309998 INFO: task ll_ost00_027:20788 blocked for more than 90 seconds. ll_ost00_027 D ffff92242eda9080 0 20788 2 0x00000080 Call Trace: schedule+0x29/0x70 wait_transaction_locked+0x85/0xd0 [jbd2] add_transaction_credits+0x278/0x310 [jbd2] start_this_handle+0x1a1/0x430 [jbd2] jbd2__journal_start+0xf3/0x1f0 [jbd2] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] osd_trans_start+0x1e7/0x570 [osd_ldiskfs] ofd_trans_start+0x75/0xf0 [ofd] ofd_attr_set+0x586/0xb00 [ofd] ofd_setattr_hdl+0x31d/0x960 [ofd] tgt_request_handle+0xb7e/0x1700 [ptlrpc] ptlrpc_server_handle_request+0x253/0xbd0 [ptlrpc] ptlrpc_main+0xc09/0x1c30 [ptlrpc] </pre> </div></div> <p>Perf stats show that a large amount of CPU time is used in preallocation:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Samples: 86M of event 'cycles', 4000 Hz, Event count (approx.): 25480688920 lost: 0/0 drop: 0/0 Overhead Shared Object Symbol 23,81% [kernel] [k] _raw_qspin_lock 21,90% [kernel] [k] ldiskfs_mb_use_preallocated 20,16% [kernel] [k] __raw_callee_save___pv_queued_spin_unlock 15,46% [kernel] [k] ldiskfs_mb_normalize_request 1,21% [kernel] [k] rwsem_spin_on_owner 0,98% [kernel] [k] native_write_msr_safe 0,54% [kernel] [k] apic_timer_interrupt 0,51% [kernel] [k] ktime_get </pre> </div></div> LU-16691

optimize ldiskfs prealloc (PA) under random read workloads

Improvement Minor Resolved Fixed Alex Zhuravlev Andreas Dilger ldiskfs Fri, 31 Mar 2023 05:01:25 +0000 Sat, 29 Jul 2023 07:57:53 +0000 Sun, 9 Jul 2023 14:25:08 +0000 Lustre 2.16.0 Lustre 2.15.2 Lustre 2.16.0 0 6 <p>Looking at the flame graphs, I would suspect that something may be wrong with the preallocation (PA), for example too many PA regions, or something else that is causing these functions to be slow. According to the flame graph <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/48613/48613_oss07.perf.svg" title="oss07.perf.svg attached to LU-16691">oss07.perf.svg^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>, for each call to <tt>ldiskfs_mb_new_blocks()</tt> there is a large amount of time spent in _raw_spin_lock(), ldiskfs_mb_normalize_request(), and ldiskfs_mb_use_preallocated().</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ldiskfs_fsblk_t ldiskfs_mb_new_blocks(handle_t *handle, struct ldiskfs_allocation_request *ar, <span class="code-object">int</span> *errp) { : : <span class="code-keyword">if</span> (!ldiskfs_mb_use_preallocated(ac)) { ac-&gt;ac_op = LDISKFS_MB_HISTORY_ALLOC; ldiskfs_mb_normalize_request(ac, ar); repeat: <span class="code-comment">/* allocate space in core */</span> *errp = ldiskfs_mb_regular_allocator(ac); </pre> </div></div> <p>so these heavy functions are <b>before</b> <tt>ldiskfs_mb_regular_allocator()</tt> is called. There is a loop in <tt>ldiskfs_mb_use_preallocated()</tt> that is repeatedly getting a spinlock, but it doesn't appear to be successful in finding a good PA, since the function ends up returning "0" and then <tt>ldiskfs_mb_normalize_request()</tt> is called anyway:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ldiskfs_mb_use_preallocated(struct ldiskfs_allocation_context *ac) { <span class="code-comment">/* first, <span class="code-keyword">try</span> per-file preallocation */</span> list_for_each_entry_rcu(pa, &amp;ei-&gt;i_prealloc_list, pa_inode_list) { : <span class="code-comment">/* found preallocated blocks, use them */</span> spin_lock(&amp;pa-&gt;pa_lock); <span class="code-keyword">if</span> (pa-&gt;pa_deleted == 0 &amp;&amp; pa-&gt;pa_free) { : <span class="code-comment">/* <span class="code-keyword">this</span> branch is never taken */</span> : <span class="code-keyword">return</span> 1; } spin_unlock(&amp;pa-&gt;pa_lock); } : /* * search <span class="code-keyword">for</span> the prealloc space that is having * minimal distance from the goal block. */ <span class="code-keyword">for</span> (i = order; i &lt; PREALLOC_TB_SIZE; i++) { list_for_each_entry_rcu(pa, &amp;lg-&gt;lg_prealloc_list[i], pa_inode_list) { spin_lock(&amp;pa-&gt;pa_lock); <span class="code-keyword">if</span> (pa-&gt;pa_deleted == 0 &amp;&amp; pa-&gt;pa_free &gt;= ac-&gt;ac_o_ex.fe_len) { cpa = ldiskfs_mb_check_group_pa(goal_block, pa, cpa); } spin_unlock(&amp;pa-&gt;pa_lock); } </pre> </div></div> <p>and then in <tt>ldiskfs_mb_normalize_request()</tt> it looks like the same PA lists are walked again and the same locks are contended:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ldiskfs_mb_normalize_request(struct ldiskfs_allocation_context *ac, struct ldiskfs_allocation_request *ar) { : list_for_each_entry_rcu(pa, &amp;ei-&gt;i_prealloc_list, pa_inode_list) { ldiskfs_lblk_t pa_end; <span class="code-keyword">if</span> (pa-&gt;pa_deleted) <span class="code-keyword">continue</span>; spin_lock(&amp;pa-&gt;pa_lock); : <span class="code-comment">/* lots of checks */</span> : spin_unlock(&amp;pa-&gt;pa_lock); } } </pre> </div></div> <p>By all rights, since these PA lists are on a single inode, there shouldn't be much contention, but it seems to fit the pattern shown by the flame graphs. Unfortunately, it isn't possible to know if the slow threads were all accessing a single file or different files.</p> <p>I think it makes sense to backport either <a href="https://patchwork.ozlabs.org/project/linux-ext4/list/?series=346731" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.ozlabs.org/project/linux-ext4/list/?series=346731</a> to ldiskfs, or at least the prealloc list fixed limit patch <a href="https://lore.kernel.org/all/d7a98178-056b-6db5-6bce-4ead23f4a257@gmail.com/" class="external-link" target="_blank" rel="nofollow noopener">https://lore.kernel.org/all/d7a98178-056b-6db5-6bce-4ead23f4a257@gmail.com/</a> to prevent the PA list from getting too long...</p> <blockquote><p> <a href="https://lore.kernel.org/all/d7a98178-056b-6db5-6bce-4ead23f4a257@gmail.com/" class="external-link" target="_blank" rel="nofollow noopener">https://lore.kernel.org/all/d7a98178-056b-6db5-6bce-4ead23f4a257@gmail.com/</a> </p></blockquote> <p>this one looks simple enough</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50481" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50481</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16691" title="optimize ldiskfs prealloc (PA) under random read workloads" class="issue-link" data-issue-key="LU-16691"><del>LU-16691</del></a> ldiskfs: limit preallocation list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: acf2f540db47d223e6999e5923aec8549be52d0b</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50481/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50481/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16691" title="optimize ldiskfs prealloc (PA) under random read workloads" class="issue-link" data-issue-key="LU-16691"><del>LU-16691</del></a> ldiskfs: limit length of per-inode prealloc list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b16c9333a00802faea419dfe6fbb013c4477c9c6</p> <p>Landed for 2.16</p> Related LU-12970 Development Rank 1|i03hq7: Rank (Obsolete) 9223372036854775807