https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12578 Lustre <p>During testing on SSD platform this was reported:</p> <p>CPU usage is in 95-100% range during obdfilter-survey. The perf trace looks very much like if <a href="https://jira.whamcloud.com/browse/LU-11347" title="Do not use pagecache for SSD I/O when read/write cache are disabled" class="issue-link" data-issue-key="LU-11347"><del>LU-11347</del></a> is not applied but it is and OSS caches are disabled.</p> <p>This is due to the lifetime of pages vs the obdfilter/obdecho lifetime:</p> <p>dt_bufs_get / osd_bufs_get does allocates pages while<br/> dt_bufs_put / osd_bufs_put marks the pages up-to-date and recycles them.<br/> This is the typical pattern for longer lived osd threads.</p> <p>For obdecho the threads are short lived and so the pattern matches<br/> init/get/put/fini meaning the survey may not be a clear representation of actual osd performance.</p> <p> </p> LU-12578

obdfilter-survey w/echo-client does not gain from direct-io optimizations

Improvement Minor Resolved Fixed Shaun Tancheff Shaun Tancheff Tue, 23 Jul 2019 15:46:54 +0000 Fri, 4 Nov 2022 22:44:27 +0000 Thu, 15 Aug 2019 13:49:22 +0000 Lustre 2.13.0 0 6 <p>why do you think that osd_bufs_get() allocates the pages? what exact version do you use?</p> <p>cray-2-12 / master</p> <p><b>osd_bufs_get</b>()<br/>   -  for (i = 0; i &lt; npages; i+<ins>, lnb</ins><ins>) { for (i = 0; i &lt; npages; i</ins><ins>, lnb</ins>+) {<br/>            lnb-&gt;lnb_page = <b>osd_get_page</b>(env, dt, lnb-&gt;lnb_file_offset,      gfp_mask);</p> <p><b>osd_get_page</b>()<br/>      if (osd_use_page_cache(d)) </p> {           // page from page cache      } <p> else </p> {          page = *alloc_page*(gfp_mask);      } <p>I'll push a patch a little while ... it's just a starting off point, feel free the tear it apart <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p> </p> <p> </p> <p>well, it's a bit different:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (unlikely(!oti-&gt;oti_dio_pages[cur])) { LASSERT(cur &lt; PTLRPC_MAX_BRW_PAGES); page = alloc_page(gfp_mask); <span class="code-keyword">if</span> (!page) <span class="code-keyword">return</span> NULL; oti-&gt;oti_dio_pages[cur] = page; </pre> </div></div> <p>oti_dio_pages stores allocated pages within @env</p> <p>For obdfilter 'env' doesn't stay around to hold onto the oti_dio_pages[] long enough ... so it doesn't get the advantage of the page pooling there.</p> <p>a perf snapshot:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 99.84% [kernel.kallsyms] | |--80.86%--native_queued_spin_lock_slowpath | | | |--75.52%--_raw_spin_lock | | | | | |--36.83%--get_page_from_freelist | | | __alloc_pages_nodemask | | | | | | | --36.73%--osd_bufs_get | | | ofd_preprw_write.isra.34 | | | ofd_preprw | | | echo_client_prep_commit.isra.52 | | | echo_client_iocontrol | | | class_handle_ioctl | | | obd_class_ioctl | | | do_vfs_ioctl | | | ksys_ioctl | | | __x64_sys_ioctl | | | do_syscall_64 | | | entry_SYSCALL_64_after_hwframe | | | __GI___ioctl | | | | | |--36.40%--free_pcppages_bulk | | | | | | | --36.40%--free_unref_page | | | | | | | --36.40%--osd_key_fini | | | key_fini | | | keys_fini.part.48 | | | lu_env_fini | | | echo_client_iocontrol | | | class_handle_ioctl | | | obd_class_ioctl | | | do_vfs_ioctl</pre> </div></div> <p>Shaun Tancheff (stancheff@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35596" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35596</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12578" title="obdfilter-survey w/echo-client does not gain from direct-io optimizations" class="issue-link" data-issue-key="LU-12578"><del>LU-12578</del></a> obdecho: Reduce CPU needed for obdfilter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6466a04bdb9b662afeeb98b33b6afbb09a8c6897</p> <p>When you write "obdfilter" in this ticket, it seems you mean "obdfilter-survey" with evho_client? It is confusing because the code that is now "ofd" was formerly called "obdfilter". </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35700/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35700/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12578" title="obdfilter-survey w/echo-client does not gain from direct-io optimizations" class="issue-link" data-issue-key="LU-12578"><del>LU-12578</del></a> obdecho: reuse an cl env cache for obdecho survey<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 55c33b70c46fde1b62f9852dc361d382a7722009</p> <p>Landed for 2.13</p> Related LU-12707 Development Rank 1|i00k2n: Rank (Obsolete) 9223372036854775807