https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3300 Lustre <p>Lustre's proc seems to have had a number of regressions. LMT's ltop is no longer able to find many of the values it used to display.</p> <p>In particular, brw_stats from obdfilter is gone, and does not appear to have been replaced after the OSD work. At minimum, that was used by ltop to report the number of bulk rpcs handled.</p> <p>The MDS display is also missing a number of values.</p> <p>We don't necessarily need to put them back exactly how they were before, but we need to export them in some way that will make them usable for folks.</p> <p>It would be best to decide on interfaces before 2.4.0 is locked in.</p> LU-3300

Restore missing proc information for LMT

Task Major Open Unresolved Emoly Liu Christopher Morrone Wed, 8 May 2013 21:49:57 +0000 Tue, 7 Jun 2016 15:38:01 +0000 Lustre 2.4.0 Lustre 2.4.0 0 10 <p><a href="https://jira.whamcloud.com/browse/LU-3296" title="fs/lustre/mdt/*/md_stats not showing any stats after remount" class="issue-link" data-issue-key="LU-3296"><del>LU-3296</del></a> is another issue related to disappearing md_stats.</p> <p>I do see brw_stats in obdfilter, but it's a symlink to osd-ldiskfs/*/brw_stats.</p> <p>Chris, the brw_stats symlink was recently added back in <a href="http://review.whamcloud.com/5873" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5873</a> (<a href="https://jira.whamcloud.com/browse/LU-3106" title="create symlinks in procfs from ofd to osd" class="issue-link" data-issue-key="LU-3106"><del>LU-3106</del></a>), so maybe that isn't in your tree yet?</p> <p>There is also a separate patch <a href="http://review.whamcloud.com/4618" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4618</a> (<a href="https://jira.whamcloud.com/browse/LU-2096" title="name ofd device type &quot;ofd&quot; instead of &quot;obdfilter&quot;" class="issue-link" data-issue-key="LU-2096">LU-2096</a>), which adds symlinks from obdfilter/ to ofd/ for some of the other tunables so that the names are cleaned up. I haven't refreshed that patch lately since it isn't really a 2.4.0 priority.</p> <p>Under ldiskfs Lustre code had driect access to the block device request queue, making brw_stats possible, as they are really just enhanced block device stats. With zfs backends there is no such access so no brw_stats.</p> <p>During proc init for the ofd device, if the underlying osd has a brw_stats file in proc we create a symlink from the obdfilter/*/ directory. If the underlying osd device does not then there will be no symlink.</p> <p>There remain stats in /proc/fs/lustre/obdfilter/lustre-OST0000/stats for bulk transfers.</p> <p>Which MDS/MDT values are you missing?</p> <p>Ah, that explains why I couldn't find brw_stats. That is a problem.</p> <p>For MDS I won't have time to figure it out before I disappear on vacation. But some of the values never show anything but zeroes. Robert's pointer about md_stats could be the problem.</p> <p>Another missing item is the per-client brw_stats on servers. That used to be our admins' main method of determining who was overloading a server when server loads went through the roof. What do we do to handle that now?</p> <p>Doh, forgot about the lack of brw_stats for ZFS. Alex's patch is only helping if brw_stats is available in the first place.</p> <p>Many of the disk brw_stats might be available on an aggregate basis, if the plumbing is available in ZFS. The per-client and per-job brw_stats is much more tricky because the ZFS IO is not allocated or submitted to disk until long after the service thread has completed processing the request. </p> <p>The RPC information like "pages per bulk r/w" and "discontiguous pages" could be available independent of the OSD type. These should really be OFD statistics, maybe in a new "rpc_stats" file, possibly in YAML format? These could also be available on a per-client or per-request basis. This might be enough for your debugging purposes?</p> <p>Information like "disk I/Os in flight", "I/O time", and "disk I/O size" might be gotten at an aggregate basis from ZFS. The "discontiguous blocks" and "disk fragmented I/Os" would be much harder to collect for writes, without deep hooks into the ZFS IO scheduler. Some of this information could be extracted for reads, by hacking into the ZFS block pointers to get the physical disk blocks.</p> <p>As for getting this into 2.4.0, I don't think that is very likely, since we are very close to making an RC1 tag. I don't think it would be unreasonable to call this a regression and fix it for 2.4.1 if it can be done cleanly.</p> <p>I don't really want to fully recreate brw_stats as it exists now. But we need a consistent way to get the same information form both ldiskfs and zfs osds, that lets us fully fill in the information that ltop has always presented.</p> <p>Proc information is important to your customers. Proc has regressed, and is incomplete. I think that means that lustre isn't ready for RCs yet.</p> <p>The obdfilter stats should provide a good view of utilization on a global and per client level, regardless of the backend. I have personally used them to this effect.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /proc/fs/lustre/obdfilter/lustre-OST0000/stats snapshot_time 1368063951.465558 secs.usecs read_bytes 7 samples [bytes] 4096 1048576 5251072 write_bytes 1287 samples [bytes] 4096 4096 5271552 get_info 8 samples [reqs] connect 1 samples [reqs] reconnect 4 samples [reqs] disconnect 1 samples [reqs] statfs 9278 samples [reqs] create 4 samples [reqs] destroy 3 samples [reqs] sync 1282 samples [reqs] preprw 1294 samples [reqs] commitrw 1294 samples [reqs] ping 9588 samples [reqs] # cat /proc/fs/lustre/obdfilter/lustre-OST0000/exports/0@lo/stats snapshot_time 1368063954.225882 secs.usecs read_bytes 7 samples [bytes] 4096 1048576 5251072 write_bytes 1287 samples [bytes] 4096 4096 5271552 get_info 8 samples [reqs] disconnect 1 samples [reqs]u create 4 samples [reqs] destroy 3 samples [reqs] sync 1282 samples [reqs] preprw 1294 samples [reqs] commitrw 1294 samples [reqs] ping 9591 samples [reqs] </pre> </div></div> <p>If something is missing then please say so. Not that I can guarantee anything for 2.4, but I would like to know.</p> <p>As an aside, if LLNL/LMT/you depend on some aspect of proc then it would be good to have some sanity tests to verify that it doesn't go away, along a comment in the test to the effect that LLNL/LMT/you will be unhappy if it does. I finally get to say that to somebody, rather than have it said to me. I thought I would enjoy it more. Weird.</p> <p>Chris, does John's proposal for using the obdfilter "stats" data address your needs for LMT? This would be a way for LMT to get aggregate IO and per-client IO stats that works on both 2.1 and 2.4.</p> <p>It would also be possible to create the brw_stats for ZFS with just the RPC ("page") information to start with, but based on your comment I don't know if this is what you want. It isn't clear to me if it will be possible to add the ZFS block IO stats later or not. Would just having the "page" information in ZFS brw_stats be useful? This would allow the admins to at least see whether the clients are submitting poorly formed RPCs. I don't think it would be too hard to do just that part.</p> <p>I discussed this with Marc Stearman today. This does not need to hold the release but it will be a support priority to find ways to enable LLNL sysadmins to perform common tasks. Marc will provide a prioritized list of those that matter most to LLNL.</p> <p>I noticed with recent 2.4 builds that lmt is failing to capture metrics on the MDS because several files are empty, however this worked in the 2.4.63 builds that I was using for my LUG testing, so this is a recent regression. These files are in both the lod and osd-ldiskfs directories and they're empty in both:</p> <blockquote> <p><span class="error">&#91;ec2-user@mds0 ~&#93;</span>$ head /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/*<br/> ==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/activeobd &lt;==<br/> 8</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/blocksize &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/desc_uuid &lt;==<br/> scratch-MDT0000-mdtlov_UUID</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/filesfree &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/filestotal &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/kbytesavail &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/kbytesfree &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/kbytestotal &lt;==</p> <p>==&gt; /proc/fs/lustre/lod/scratch-MDT0000-mdtlov/numobd &lt;==<br/> 8</p></blockquote> <p>Robert, I broke the osd statfs handlers. A patch is forthcoming.</p> <p>Please see <a href="http://review.whamcloud.com/6385" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6385</a>.</p> <p>The patch for the osd statfs proc handlers has landed to master. But I'm leaving this ticket open waiting for a response from LLNL.</p> <p>With that patch landed I'm getting metadata in LMT again, thanks! Still waiting to hear from LLNL...</p> <p>Chris, any updates here? We were waiting on a response from LLNL.</p> <p>The problem that you fixed we hadn't even seen yet. We did then see it. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Now we don't.</p> <p>So now brw_stats, or something equivalent just needs to be added, I believe.</p> <p>Emoly</p> <p>Could you please see what work remains on this ticket?</p> <p>Thanks</p> <p>Peter</p> <p>Chris, I add a symlink of osd brw_stats to lod. Hope that can meet your requirement.<br/> Please see <a href="http://review.whamcloud.com/7663" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7663</a> .</p> <p>Chris, is there any more work you would like us to do for this ticket? Or can we mark it as resolved? Thanks.</p> Duplicate LU-3106 Related LU-2396 LU-3296 LU-2096 LU-4259 LU-3355 LU-4259 Development End date Mon, 29 Feb 2016 21:49:57 +0000 Rank 1|hzvqh3: Rank (Obsolete) 8169 Start date Wed, 8 May 2013 21:49:57 +0000