https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15642 Lustre <p>The patch <a href="https://review.whamcloud.com/46075" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46075</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12585" title="obdfilter read_bytes stat reports requested read bytes and not actual read bytes" class="issue-link" data-issue-key="LU-12585"><del>LU-12585</del></a> obdfilter: Use actual I/O bytes in stats</tt>" changed the measurement of the read/write latency stats, so that they now contain the network round-trip time, while they previously only contained the local filesystem IO time. This caused the reported latency with 46076 applied to be much higher (tens or hundreds of <em>milli</em>seconds) than the pre-patch latency (hundreds of <em>micro</em>seconds for flash).</p> <p>While it may be necessary to account for the actual read/write bytes after the RPC transfer is complete, the code should account for the IO latency after the IO is complete, as it did before, rather than after the RPC is complete. The RPC stats at the OST level and on the client will include the full RPC latency, and the ofd stats should only account for the storage latency.</p> LU-15642

restore server read/write latency measurements

Bug Minor Open Unresolved Patrick Farrell Andreas Dilger Sat, 12 Mar 2022 05:11:02 +0000 Thu, 11 May 2023 00:42:45 +0000 Lustre 2.15.0 0 4 <p>&gt; While it may be necessary to account for the actual read/write bytes after the RPC transfer is complete, the code should account for the IO latency after the IO is complete, as it did before, rather than after the RPC is complete. The RPC stats at the OST level and on the client will include the full RPC latency, and the ofd stats should only account for the storage latency.</p> <p> </p> <p>I don't know how much work this is, or if this is the best way to do things, but I think it might be useful to have both sets of counters (including for metadata operations):</p> <ul> <li>I/O completion</li> <li>I/O completion + after RPC completion (&lt;countername&gt;_rtt, or maybe an extra field that needs to be enabled via a tunable?)</li> </ul> <p>You can certainly collect the per client counters (via llite), but it's a lot more difficult to collect/munge/etc all of the client data than having an overall average server side for general use, such as identifying network congestion outside of the Lustre servers' control. </p> <p>Steve, there are already stats that include the RPC network transfer time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param obdfilter.*.stats| egrep "read|write|=" obdfilter.testfs-OST0003.stats= read_bytes 26 samples [bytes] 1048576 4194304 104857600 433207581343744 write_bytes 25 samples [bytes] 4194304 4194304 104857600 439804651110400 read 26 samples [usecs] 269 34432 83969 2362054419 write 25 samples [usecs] 546 23458 87802 1197733848 # lctl get_param ost.OSS.ost_io.stats | egrep "read|write" ost_read 26 samples [usec] 2238 101196 469557 22435172467 ost_write 25 samples [usec] 6709 106630 774811 36953324201 </pre> </div></div> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46833</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15642" title="restore server read/write latency measurements " class="issue-link" data-issue-key="LU-15642">LU-15642</a> obdclass: use consistent stats units<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a171d269253076d7c6c3e827ce116cab142add83</p> <p>Note above patch does not fix the IO latency stats, just some improvements while I was looking at this code.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46833/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46833/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15642" title="restore server read/write latency measurements " class="issue-link" data-issue-key="LU-15642">LU-15642</a> obdclass: use consistent stats units<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b515c6ec2ab84598c77c65eb78f1afd5e67b1ede</p> Related LU-16231 LU-12585 LU-16110 Development Rank 1|i02kq7: Rank (Obsolete) 9223372036854775807 Severity