https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10698 Lustre <p>One thing that Lustre has been missing for a long time is I/O profiling. Lustre does support I/O profiling per process and per client, but it doesn't support profiling I/O per job, which is the commonly used case in reality.</p> <p>It would be desirable to add I/O profiling to client, llite in particular, by <tt>JOBID</tt>, also it will be better to provide tools to accumulate those stats from multiple clients and plot them correspondingly.</p> LU-10698

Specify complex JobIDs for Lustre

Improvement Major Resolved Fixed Jinshan Xiong Jinshan Xiong Thu, 22 Feb 2018 05:08:52 +0000 Wed, 7 Feb 2024 19:27:34 +0000 Fri, 31 Aug 2018 07:34:13 +0000 Lustre 2.12.0 0 10 <p>I know darshan is able to profile I/O by intercepting glibc callbacks but still it would be better if Lustre can support it natively.</p> <p>Jinshan Xiong (jinshan.xiong@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/31500" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31500</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10698" title="Specify complex JobIDs for Lustre" class="issue-link" data-issue-key="LU-10698"><del>LU-10698</del></a> obdclass: cleanup jobid implementation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3ef0ceb1c27fb02ec8c93b53333365d3fb36cd27</p> <p>I can understand that if hundreds of nodes are generating unlabelled RPCs then using <tt>procname_uid</tt> could result in a lot of "rsync.1234", "rsync.2345", "ls.5678", "cp.9876", etc. kind of results if there are many active users, but otherwise this still provides useful information about what commands are generating a lot of IO traffic. The reason "<tt>procname.uid</tt>" was chosen as the fallback if <tt>JOBENV</tt> can't be found is that there is a good likelihood of the same user running on different nodes without an actual JobID to still generate the same jobid string, unlike embedding PID or other unique identifier (which would be useless after the process exits anyway).</p> <p>One option would be to allow userspace to specify a fallback jobid if obd_jobid_var is not found. This could be a more expressive syntax for the primary/fallback than just "<tt>disabled</tt>", "<tt>procname_uid</tt>", and "<tt>nodelocal</tt>" that can be specified today. For example interpreting "<tt>%proc.%uid</tt>" as "<tt>process name</tt>" '<tt>.</tt>' "<tt>user id</tt>", but allowing just "<tt>%proc</tt>", just "<tt>%uid</tt>", but also maybe "<tt>%gid</tt>", "<tt>%nid</tt>", "<tt>%pid</tt>", and other fields as desired (filtering out any unknown '%' and other escape characters). This could instead use a subset of escapes from core filenames in <tt>format_corename()</tt>, to minimize the effort for sysadmins (e.g. <tt>%e</tt>=executable, <tt>%p</tt>=PID (and friends?), <tt>%u</tt>=UID, <tt>%g</tt>=UID, <tt>%h</tt>=hostname, <tt>%n</tt>=NID). It isn't clear to me yet if PID is useful for JobID, but it isn't hard to implement and maybe there is a case for it.</p> <p>Unknown strings would just be copied literally, so you could set:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lctl set_param jobid_var=PBS_JOBID lctl set_param jobid_name='%e.%u:%g_%n' </pre> </div></div> <p>or to get Jinshan's desired behaviour just set:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lctl set_param jobid_name='unknown' </pre> </div></div> <p>This implies that if "<tt>JOBENV</tt>" is not found then "<tt>jobid_name</tt>" would be used as a fallback (which doesn't happen today), and would be interpreted as needed.</p> <p>Using "<tt>jobid_var=nodelocal</tt>" would keep "<tt>jobid_name</tt>" as a literal string as it is today, while allowing the kernel to generate useful jobids directly, similar to core dump filenames. My preference would be to keep "<tt>jobid_name=%e.%u</tt>" as the default if jobstats is enabled, since this is what we currently have, and is at least providing some reasonable information to users that didn't set anything in advance.</p> <p>I'm not a big fan of reinventing printf for just jobids. We had a similar proposal within Cray for <a href="https://jira.whamcloud.com/browse/LU-9789" title="Create JobID prefix" class="issue-link" data-issue-key="LU-9789"><del>LU-9789</del></a> and it didn't ever get implemented due to complexity and that a simple "good enough" solution would work for pretty much everyone.</p> <p>Hi Andreas,</p> <p>It seems to be a good suggestion but probably won't be accomplished in short time.</p> <p>One thing I want to clarify is that providing too much information is not necessary good. For example, actions like collecting and saving the status of all disk drives in a cluster are not good at all because useful information are completely flooded. We should only care about the situations that some components are not working properly, like some OSTs are in degraded mode, and it should be a separate procedure to figure out which drives are not working.</p> <p>So in this case if some workload are running without proper jobid setting, I tend to think it's not a good practice to fallback to 'procname.uid' because:<br/> 1. it may be difficult to extract useful information from a very long list of stats;<br/> 2. if anonymous workload can be accumulated to a single entry, it's easier to know how much resource it's consumed. If it's little, probably it will be simply ignored; otherwise a separate procedure will be performed to figure out which anonymous jobs consumed that much resource.</p> <p>I hope this will make some sense.</p> <p>Ben, at the same time, the proposed "cluster ID" functionality could be implemented in a similar manner rather than adding a special-case handler for the cluster. Something like <tt>jobid_name="clustername.%j"</tt> since the cluster name will be constant for the lifetime of the node and can just be set as a static string from the POV of the kernel. </p> <p>I don't think the implementation would be too complex, basically a scan for '%' in the string, then a switch statement that replaces the string with a known value (length limited to output buffer string). </p> <p>Jinshan, as for dumping all unknown RPCs into a single bucket, that is OK if they don't take up much of the resource, but as you write then more work is needed if it <em>does</em> take up a lot of the resources, so it would be useful to have a way to debug that. Your'ee replacing the case that works well with Cray, but not well for you with one that works for you but not Cray (and IMHO will work badly for you as soon as you want to debug what is causing a lot of "unknown" traffic). I think we can have a solution that works for both of you that doesn't add too much complexity. </p> <blockquote> <p>... so it would be useful to have a way to debug that. </p></blockquote> <p>True, in that case, the admin should clear <tt>job_stats</tt>, and set the <tt>jobid_var</tt> to <tt>procname_uid</tt> and then monitor the output of <tt>job_stats</tt> in real-time manner to figure out who the 'bad' guy is.</p> <p>It boils down to the point if <tt>job_stats</tt> is mainly for monitoring or auditing. Cray's customer would like to use it as monitoring, but I think we should use it for auditing. Obviously we don't work for the same customer lol.</p> <p>Jinshan, if this is something you care about in a database, simply pre-process it on insertion to ignore procname.uid style entries.</p> <p>If, on the other hand, you want this information, it can't be generated from "unknown"</p> <p>Hi Evans,</p> <p>Not just discard procname.uid records, we also need to accumulate them because we want to know how much I/O is from anonymous therefore we can decide if to start investigation.</p> <p>Can you please summarize how you and your customers use job_stats information? So it sounds like those data will be kept in memory and never collected?</p> <p>Jinshan, all of what you propose can be done in userspace. You can translate all procname.uid formatted JobID's to "unknown", you can leave them out of the database you use for mining. What you can't do, is take stats from Lustre of "Unknown" and translate them into "rsync.12345" on 6 different nodes.</p> <p>My understanding from what I've seen from the management side of our Lustre products is that they are accumulating each job, and scoring it in a number of ways, along with keeping it in a database for deeper investigation. I'm not sure what the limits may be concerning what is kept in the DB and for how long, and at what timescales.</p> <p>I do know that this is an area of active development, as the performance penalties incurred by JobID are not as harsh as they used to be due to the cache. So we've moved from a case where JobID is off by default to one where it can be on by default.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31691" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31691</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10698" title="Specify complex JobIDs for Lustre" class="issue-link" data-issue-key="LU-10698"><del>LU-10698</del></a> obdclass: allow specifying complex jobids<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5a7b3fd923cec7d4acc199b9f205b3ea8483c495</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31691/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31691/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10698" title="Specify complex JobIDs for Lustre" class="issue-link" data-issue-key="LU-10698"><del>LU-10698</del></a> obdclass: allow specifying complex jobids<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6488c0ec57de2d188bd15e502917b762e3a9dd1d</p> Duplicate LU-9789 Related LU-17512 LUDOC-381 LU-12330 LUDOC-310 Development Rank 1|hzzt5j: Rank (Obsolete) 9223372036854775807