https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13597 Lustre <p>It would be useful to add request processing times/latency and basic IO size information to the <tt>job_stats</tt> output, similar to <tt>brw_stats</tt> and other <tt>stats</tt> files.</p> <p>This would allow monitoring per-job request processing performance, to allow differentiation between applications that are doing poorly-formed IO vs. jobs that may be suffering because of other jobs.</p> LU-13597

add processing time/latency, IO sizes, stats to job_stats

Improvement Minor Resolved Fixed Emoly Liu Andreas Dilger Sat, 23 May 2020 01:56:45 +0000 Fri, 11 Jun 2021 23:56:57 +0000 Tue, 23 Jun 2020 13:01:10 +0000 Lustre 2.14.0 0 4 <p>Emoly</p> <p>Could you please look into this?</p> <p>Thanks</p> <p>Peter</p> <p>Emily, implementing this should be similar to the change in patch <a href="https://review.whamcloud.com/36078" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36078</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12631" title="Report latency of client operations" class="issue-link" data-issue-key="LU-12631"><del>LU-12631</del></a> llite: report latency for filesystem ops</tt>" that changed over to using <tt>LPROCFS_TYPE_LATENCY</tt> for the client stats. </p> <p>We should already be tracking the processing times already in the OSS/MDS code, but they just need to be added into the per-job stats. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>,</p> <p>I saw that your patch at <a href="https://review.whamcloud.com/#/c/33201/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33201/</a> had already added time information to ***_stats structures. Should I make this one based on that one?</p> <p>BTW, our current job_stats output is like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>obdfilter.lustre-OST0000.job_stats= job_stats: - job_id: cp.0 snapshot_time: 1591005135 read_bytes: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0 } write_bytes: { samples: 1, unit: bytes, min: 1689, max: 1689, sum: 1689 } getattr: { samples: 0, unit: reqs } setattr: { samples: 0, unit: reqs } punch: { samples: 0, unit: reqs } sync: { samples: 0, unit: reqs } destroy: { samples: 0, unit: reqs } create: { samples: 0, unit: reqs } statfs: { samples: 0, unit: reqs } get_info: { samples: 0, unit: reqs } set_info: { samples: 0, unit: reqs } quotactl: { samples: 0, unit: reqs } </pre> </div></div> <p>Besides adding time information, using LPROCFS_TYPE_LATENCY for all the requests, and anything else?</p> <p>The <tt>read_bytes</tt> and <tt>write_bytes</tt> would stay with <tt>bytes</tt> units, but the others would change to <tt>usec</tt> units and show min/max/sum values. It would also be good to add a separate <tt>usec</tt> counter for the <tt>read</tt> and <tt>write</tt> operations, so that it is possible to calculate bandwidth. </p> <p>The job_stats from ofd have been changed. The output is like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- job_id: cp.0 snapshot_time: 1591174981 read_bytes: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0, sumsq: 0 } write_bytes: { samples: 1, unit: bytes, min: 2765, max: 2765, sum: 2765, sumsq: 7645225 } read_time: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } write_time: { samples: 1, unit: usecs, min: 29, max: 29, sum: 29, sumsq: 841 } getattr: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } setattr: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } punch: { samples: 1, unit: usecs, min: 49, max: 49, sum: 49, sumsq: 2401 } sync: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } destroy: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } create: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } statfs: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } get_info: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } set_info: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } quotactl: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } prealloc: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } </pre> </div></div> <p>Do the ones from mdt need to be changed too?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt.lustre-MDT0000.job_stats= job_stats: - job_id: cp.0 snapshot_time: 1591175698 open: { samples: 1, unit: reqs } close: { samples: 1, unit: reqs } mknod: { samples: 1, unit: reqs } link: { samples: 0, unit: reqs } unlink: { samples: 0, unit: reqs } mkdir: { samples: 0, unit: reqs } rmdir: { samples: 0, unit: reqs } rename: { samples: 0, unit: reqs } getattr: { samples: 1, unit: reqs } setattr: { samples: 0, unit: reqs } getxattr: { samples: 1, unit: reqs } setxattr: { samples: 0, unit: reqs } statfs: { samples: 0, unit: reqs } sync: { samples: 0, unit: reqs } samedir_rename: { samples: 0, unit: reqs } crossdir_rename: { samples: 0, unit: reqs } read_bytes: { samples: 0, unit: reqs, min: 0, max: 0, sum: 0 } write_bytes: { samples: 0, unit: reqs, min: 0, max: 0, sum: 0 } punch: { samples: 0, unit: reqs } migrate: { samples: 0, unit: reqs } </pre> </div></div> <p>Emoly Liu (emoly@whamcloud.com) uploaded a new patch:<a href="https://review.whamcloud.com/38816" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38816</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13597" title="add processing time/latency, IO sizes, stats to job_stats" class="issue-link" data-issue-key="LU-13597"><del>LU-13597</del></a> ofd: add more information to job_stats<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a9aff2fad827da1b1dc521ed29b001f003a4b30d</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> write_bytes: { samples: 1, unit: bytes, min: 2765, max: 2765, sum: 2765, sumsq: 7645225 } write_time: { samples: 1, unit: usecs, min: 29, max: 29, sum: 29, sumsq: 841 } punch: { samples: 1, unit: usecs, min: 49, max: 49, sum: 49, sumsq: 2401 } </pre> </div></div></blockquote> <p>This looks good, writing about 95MB/s...</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> read_bytes: { samples: 0, unit: reqs, min: 0, max: 0, sum: 0 } write_bytes: { samples: 0, unit: reqs, min: 0, max: 0, sum: 0 } </pre> </div></div></blockquote> <p>This looks like a pre-existing bug - it should be units of "bytes" and not "reqs".</p> <p>In any case, the MDS jobstats should also be updated to report usec.</p> <p>The job_stats output of MDT is like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- job_id: touch.500 snapshot_time: 1591267859 open: { samples: 1, unit: usecs, min: 146, max: 146, sum: 146, sumsq: 21316 } close: { samples: 1, unit: usecs, min: 17, max: 17, sum: 17, sumsq: 289 } mknod: { samples: 1, unit: usecs, min: 124, max: 124, sum: 124, sumsq: 15376 } link: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } unlink: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } mkdir: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } rmdir: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } rename: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } getattr: { samples: 1, unit: usecs, min: 8, max: 8, sum: 8, sumsq: 64 } setattr: { samples: 1, unit: usecs, min: 27, max: 27, sum: 27, sumsq: 729 } getxattr: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } setxattr: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } statfs: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } sync: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } samedir_rename: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } crossdir_rename: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } read_bytes: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0, sumsq: 0 } write_bytes: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0, sumsq: 0 } punch: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } migrate: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } </pre> </div></div> <p>Since the unit is changed from reqs to usecs, sometimes the sumsq looks big.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>setattr: { samples: 1, unit: usecs, min: 15492, max: 15492, sum: 15492, sumsq: 240002064 } </pre> </div></div> <p> </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38816/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38816/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13597" title="add processing time/latency, IO sizes, stats to job_stats" class="issue-link" data-issue-key="LU-13597"><del>LU-13597</del></a> ofd: add more information to job_stats<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cd8fb1e8d300c0622200d6b25d187d8d0472e79f</p> <p>Landed for 2.14</p> Related LU-12631 LU-13716 LU-13733 LU-7909 LU-11407 LU-12872 Development Rank 1|i0115b: Rank (Obsolete) 9223372036854775807