https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10777 Lustre <p>FIO test does a lot more RPCs to the MDT with DOM files comparing with OSTs files:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">DOM files: WRITE: BW 5900KiB/sec, IOPS 737, lat (7/83624/83631)usec ----- MDC RPCs: 118696 ----- OSC RPCs: 252 OST files: WRITE: BW 9738KiB/sec, IOPS 1217, lat (5/47728/47734)usec ----- MDC RPCs: 49668 ----- OSC RPCs: 12302 </pre> </div></div> <p>so there are about 12K IO-related RPCs to the OST and about 70K extra RPCs to the MDT in the same test. That looks like OST writes are cached much better with FIO write pattern than writes to MDT.</p> LU-10777

DoM performance is bad with FIO write

Technical task LU-10176 Major Resolved Fixed Mikhail Pershin Mikhail Pershin DoM2 ORNL Tue, 6 Mar 2018 08:59:19 +0000 Sat, 8 Jun 2019 12:44:27 +0000 Tue, 30 Apr 2019 12:57:09 +0000 Lustre 2.13.0 Lustre 2.12.3 0 8 <p>there are smaller page-per-RPC value with DOM files, the write RPC is composed earlier, with less pages than with OST. The problem is related to grants most likely.</p> <p>This issue doesn't look like a bug but related to less grants space on MDT comparing with several OSTs. With increasing MDT size the page-per-RPC value is growing and performance as well.</p> <p>Meanwhile there is other problem with FIO READ which is related to read-on-open feature and possible resends if reply buffer is bigger than client had. In that case client re-allocated buffer and resends it with larger buffer. For some reason such technique performs not well and even worse than just separate OPEN and READ RPCs. I am going to disable that option for now</p> <p>Is this happening on the case of intent-open?</p> <p>Can we just do a quick fix to fallback to normal open if client doesn't allocate buffer with enough size?</p> <p>that is so already, read-on-open failure doesn't fail open itself, just doesn't fill reply buffer with data and finish open intent as usual.</p> <p>patch in gerrit: <a href="https://review.whamcloud.com/#/c/34700/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34700/</a></p> <p>If that's the case, why would it resend the RPC if client provides a small reply buffer, instead of just doing a regular open? or I have missed something here.</p> <p>ah I see, you patch does what I said. Nevermind</p> <p>That is being handled on ptlrpc level, check reply_in_callback() and after_reply() for rq_reply_truncate usage. If reply has bigger buffer than allocated on client then client re-allocate it and do resend with adjusted buffer with no errors are returned to upper level, e.g. MDC. That is used in some cases intentionally, e.g. when big attributes are returned from server to the client and it has not updated 'max_md_size'. Usually these are rare cases because finally client will update own maximum.<br/> The idea behind this is that technically immediate resend is faster than new RPC or at least the same. I was trying to use that possibility for read-on-open to fill as much data as maximum reply buffer can be, thinking that resend is at least the same as new RPC, but after test results I tend to think that OPEN resend/reconstruct with new read-on-open is just more expensive than ordinary READ RPC so that makes no sense to do so.</p> <p>oh, I wrote so much words, let them be, maybe someone will find that useful</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34700/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34700/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10777" title="DoM performance is bad with FIO write" class="issue-link" data-issue-key="LU-10777"><del>LU-10777</del></a> dom: disable read-on-open with resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e0adb618a4b0d0182419a5731fe046e9157b9f51</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34912" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34912</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10777" title="DoM performance is bad with FIO write" class="issue-link" data-issue-key="LU-10777"><del>LU-10777</del></a> dom: disable read-on-open with resend<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7f304a9c1328518be315a96a2837e3452e6221d3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34912/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34912/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10777" title="DoM performance is bad with FIO write" class="issue-link" data-issue-key="LU-10777"><del>LU-10777</del></a> dom: disable read-on-open with resend<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: bfe06da80e8cb75667d0a58a8e81043a09868112</p> Development Rank 1|hzztx3: Rank (Obsolete) 9223372036854775807