https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12872 Lustre <p>JOBSTATS has been very useful to understand what type of IO is coming from application per JOBID/UID and GID, but it also would be nice to have more stats (e.g. like RPC size, discontiguous pages, etc that is covered by "brw_stats" today) into JOBSTATS to understand detail IO workload/size per JOBID.<br/> The other hand, the hybrid system with SSD and HDD (with/without same namespace) is naturally coming and those new stats allow let administrator or users know more guideline/information e.g. which OST devices are preferred by each application.</p> <p><a href="https://jira.whamcloud.com/browse/LU-12872" title="Adding more stats into JOBSTATS" class="issue-link" data-issue-key="LU-12872"><del>LU-12872</del></a> is used to track the requirement of adding more stats.</p> <p><a href="https://jira.whamcloud.com/browse/LU-16087" title="show distribution information of jobstats" class="issue-link" data-issue-key="LU-16087"><del>LU-16087</del></a> is used to track the requirement of show distribution information of jobstats.</p> LU-12872

Adding more stats into JOBSTATS

Improvement Minor Resolved Duplicate Feng Lei Shuichi Ihara Thu, 17 Oct 2019 07:39:34 +0000 Mon, 24 Apr 2023 18:57:42 +0000 Mon, 24 Apr 2023 18:56:31 +0000 0 10 <p><del>"Feng, Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/48238" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48238</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-12872" title="Adding more stats into JOBSTATS" class="issue-link" data-issue-key="LU-12872"><del>LU-12872</del></a> lprocfs: add rpc nubmer and size to job stats</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 0b8b8d76a8072a58474fc6176850b331e4cb05f8</del></p> <p>Shuichi, can you please confirm, but I think the request here is to include the same or similar information from <tt>osd-ldiskfs.<b>.brw_stats</tt> into the <tt>obdfilter.</b>.job_stats</tt>. My understanding is that the "RPC size" you requested is the bulk data size, and not the size of the RPCs themselves?</p> <p>Feng Lei, it is important to note that unlike some network filesystems, most Lustre bulk read/write RPCs sent from the clients do not contain the actual data, but only a description of the object being read/written and offsets and byte counts for each fragment in the request. Only in a few cases when the read/write request is very small is the data packed directly into the RPC. Normally, for larger RPCs (anything over 4KB), only when the server is processing a bulk read/write RPC it will set up the RDMA descriptors for the request, and then the IB HCA will transfer the data directly from client memory to server memory. That avoids the server memory being filled by the data for the requests that are queued, and avoids copying data from the network request buffers to the filesystem pages for IO.</p> <p>To fit into the <tt>job_stats</tt> file, it would need to add several sub items for each of the job_stats entries:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>obdfilter.testfs-OST0001.job_stats= job_stats: - job_id: grep.0 snapshot_time : 4562769.380032450 secs.nsecs start_time : 4562769.053337605 secs.nsecs elapsed_time : 0.326694845 secs.nsecs read_bytes: { samples: 5, unit: bytes, min: 32768, max: 4194304, sum: 16777216, sumsq: 70096013754368 } write_bytes: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0, sumsq: 0 } read: { samples: 5, unit: usecs, min: 136, max: 4072, sum: 11871, sumsq: 36286067 } write: { samples: 0, unit: usecs, min: 0, max: 0, sum: 0, sumsq: 0 } : brw_stats: { pages_per_bulk: { 8: nnn 16: mmm 32: ppp ... } discontig_pages: { } discontig_blocks: { } disk_fragments: { } disk_io_inflight: { } disk_io_time: { } disk_io_size: { } } </pre> </div></div> <p>I would like that this output is still reasonably easily read by humans, though it should also be properly parsable, unlike the current <tt>brw_stats</tt>.</p> <p>However, Joe or someone more familiar with YAML parsing than I am should provide the actual layout so that the existing <tt>job_stats</tt> parser does not explode if it sees these new fields.  In <a href="https://jira.whamcloud.com/browse/LU-13123" title="Add list of client NIDs to job_stats output" class="issue-link" data-issue-key="LU-13123">LU-13123</a> I would also like to add a list of client NIDs that sent RPCs for this JobID so that it can be isolated to a specific clients (without the need to access an external job scheduler), so this should also be taken into consideration.</p> <p>Yes, we want to see RPC/IO size for bulk IO to OSTs in Jobstats (e.g. per JOBID, UID) even per NID would be useful since such stats doesn't exist in NID "export" stats today.</p> <p>I'm going to add:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">enum</span> {     LPROCFS_CNTR_EXTERNALLOCK    = 0x0001,     LPROCFS_CNTR_AVGMINMAX      = 0x0002,     LPROCFS_CNTR_STDDEV        = 0x0004, LPROCFS_CNTR_HISTGRAM = 0x0008, ... LPROCFS_CNTR_RPC_READ_PAGES = LPROCFS_TYPE_PAGES | LPROCFS_CNTR_HISTGRAM,</pre> </div></div> <p>The output may be like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rpc_read_pages: { samples: 0, unit: pages, min: 0, max: 0, sum: 0, sumsq: 0, histgram: {1: xxx, 2: yyy, 4: zzz, ...} } </pre> </div></div> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a> Please feel free to comment.</p> <p> </p> <p>Move this topic to <a href="https://jira.whamcloud.com/browse/LU-16087" title="show distribution information of jobstats" class="issue-link" data-issue-key="LU-16087"><del>LU-16087</del></a>.</p> <p>One thing I realized is that job_stats are tracked at the ofd level, while brw_stats are tracked at the osd level because they contain on-disk allocation information. </p> <p>That means the current job_stats file could add the RPC pages_per_bulk, latency, and discontiguous_pages histograms, but not discontiguous disk blocks ir disk IO size. I think that is probably OK, since we can check the disk fragmentation from the main brw_stats (these should not be specific to the application.</p> <p>Was handled by <a href="https://jira.whamcloud.com/browse/LU-16087" title="show distribution information of jobstats" class="issue-link" data-issue-key="LU-16087"><del>LU-16087</del></a></p> Related LU-13123 LU-11407 LU-13597 LU-16087 Development Rank 1|i00o6f: Rank (Obsolete) 9223372036854775807