https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16766 Lustre <p>Reduce the long kernel thread names like "<tt>kworker/CPU:ID</tt>" to just "kworker", and "<tt>ll_sa_PID</tt>" to "<tt>ll_sa</tt>", since it is actually less useful to have the full kernel thread ID instead of aggregating these into a single process name on the stats. There may be other similar kernel thread names that should be abbreviated.</p> <p>Also, for statahead and similar Lustre threads that are generating RPCs on behalf of user processes, they should be properly accounted to the user/client. </p> LU-16766

Combine some kernel process names for jobid

Improvement Minor Resolved Fixed Thomas Bertschinger Andreas Dilger lug23dd Mon, 24 Apr 2023 19:21:45 +0000 Wed, 7 Feb 2024 19:26:38 +0000 Thu, 31 Aug 2023 15:11:39 +0000 Lustre 2.16.0 Lustre 2.16.0 0 6 <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51919" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51919</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16766" title="Combine some kernel process names for jobid" class="issue-link" data-issue-key="LU-16766"><del>LU-16766</del></a> obdclass: trim kernel thread names in jobids<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ce986c8cd5a7ef53819a82cf4d38fdfd4f532a4e</p> <p>I've uploaded a patch in progress for this but wanted to ask some design questions that may be broader than the patch I've submitted for this issue.</p> <p>First, the description here has:</p> <blockquote><p>statahead and similar Lustre threads ... should be properly accounted to the user/client.</p></blockquote> <p>It looks like <a href="https://jira.whamcloud.com/browse/LU-16781" title="account jobids to original process" class="issue-link" data-issue-key="LU-16781">LU-16781</a> is for this issue so I think this can be handled with a patch on that ticket.</p> <p><a href="https://jira.whamcloud.com/browse/LU-16781" title="account jobids to original process" class="issue-link" data-issue-key="LU-16781">LU-16781</a> also says:</p> <blockquote><p>For jobid_name_is_valid() we may consider to reduce the exclusions for kernel processes. Firstly, this exclusion doesn't always working properly, since kworker and ll_sa tasks still show up<br/> in the server stats. Secondly, this hides the real presence of RPCs sent to the server, so job_stats are not showing the full picture of what is generating the load</p></blockquote> <p>I have some questions about the reasoning behind jobid_name_is_valid(), which could be relevant to this patch.</p> <p>It looks like that check is only called if obd_jobid_var refers to an actual environment var and not one of the special values (nodelocal, session, procname_uid). Why is this the only case where the exclusion matters?</p> <p>To me it seems that the decision of whether a kernel thread should be included in jobstats is independent of whether the jobid setting is an env var, per-session var, or anything else. But I may not have the full picture here.</p> <p>Can you clarify what the intent of the exclusion is?</p> <p>If the purpose of the exclusion is just to avoid checking the process's environment in the event the process is a kernel thread,<br/> would it be more direct to replace the check for a hardcoded list of names with a check for <tt>current-&gt;flags &amp; PF_KTHREAD</tt>? Or would that be overly broad?</p> <p>The purpose of <tt>jobid_name_is_valid()</tt> is to avoid using the procname/environment from those threads when generating the jobid, and instead get this information from the inode that is being processed. Also, there are some "housekeeping" RPCs like pings that are excluded since they might otherwise flood the server logs. I <em>suspect</em> there are still a couple of bugs in how the jobid name is generated, and we <em>should</em> be using the application process jobid that was stored in the file inode for the kernel threads to use, but somehow this is not happening correctly in all cases.</p> <p>I think <tt>PF_KTHREAD</tt> is overly broad to deny generating <b>any</b> jobid for an RPC, since ptlrpcd is a kernel thread and it is generating many client RPCs. However, it may be that <tt>PF_KTHREAD</tt> is a good indicator that we shouldn't be generating the jobid from the current thread, but rather from the file of interest in the RPC...</p> <p>This is unfortunately a bit vague, since it has been some time since I was debugging this code. It might be possible to run with "<tt>+rpctrace</tt>" debugging enabled on the client and see what RPCs are being generated by <tt>kworker</tt> and what they can use to generate a better jobid for the RPC.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51919/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51919/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16766" title="Combine some kernel process names for jobid" class="issue-link" data-issue-key="LU-16766"><del>LU-16766</del></a> obdclass: trim kernel thread names in jobids<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a9c503c002d08d0587894a748761e30c1b9a445</p> <p>Landed for 2.16</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53904" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53904</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16766" title="Combine some kernel process names for jobid" class="issue-link" data-issue-key="LU-16766"><del>LU-16766</del></a> llite: fix JobID for readahead threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c213a61f4a9d1e89fa0fd15fdf85a1954c65b22a</p> Related LU-16765 LU-16767 LU-16781 LU-17512 Development Rank 1|i03jp3: Rank (Obsolete) 9223372036854775807