https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-694 Lustre <p>This feature is to collect filesystem operation stats for the jobs running on Lustre.</p> <p>When some job sheculer (SLURM, for instance) is running on lustre client, the lustre client will pack the job id into each request (open, unlink, write...), and server will collect those information then expose them via procfs.</p> LU-694

Job Stats

Improvement Minor Resolved Fixed Niu Yawei Niu Yawei Wed, 21 Sep 2011 00:31:48 +0000 Sat, 22 Nov 2014 20:04:16 +0000 Mon, 4 Jun 2012 06:07:32 +0000 Lustre 2.3.0 Lustre 2.3.0 0 13 <ul class="alternate" type="square"> <li>Enable/Disable Jobstats feature</li> </ul> <p> The Jobstats is disabled by default, that can be verified by checking the /proc/fs/lustre/jobid_var on client,<br/> the'jobid_var' should be 'disable' by default.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl get_param jobid_var jobid_var=disable </pre> </div></div> <p> To enable the Jobstats, one can specify the 'jobid_var' for a certain job scheduler. </p> <ul class="alternate" type="square"> <li>Configure 'jobid_var' for specified job scheduler</li> </ul> <p> To enable Jobstats for certain job scheduler, the 'jobid_var' should be configured as proper value:<br/> SLURM: jobid_var=SLURM_JOB_ID<br/> SGE: jobid_var=JOB_ID<br/> LSF: jobid_var=LSB_JOBID<br/> Loadleveler: jobid_var=LOADL_STEP_ID<br/> PBS: jobid_var=PBS_JOBID<br/> Maui/MOAB: jobid_var=PBS_JOBID</p> <p> For example, to enable Jobstats for SLURM on a fs named 'lustre':</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl conf_param testfs.sys.jobid_var=SLURM_JOB_ID </pre> </div></div> <p> Disable Jobstats on a fs named 'lustre':</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl conf_param testfs.sys.jobid_var=disable </pre> </div></div> <p> If there isn't any job scheduler is running over the system, or user just want to collect the stats for process &amp; uid:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl conf_param testfs.sys.jobid_var=procname_uid </pre> </div></div> <ul class="alternate" type="square"> <li>Check Job stats</li> </ul> <p> The metadata operation stats is collected on MDT, and one can access it via <tt>lctl get_param mdt.&#42;.job_stats</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl get_param mdt.lustre-MDT0000.job_stats job_stats: - job_id: bash.0 snapshot_time: 1352084992 open: { samples: 2, unit: reqs } close: { samples: 2, unit: reqs } mknod: { samples: 0, unit: reqs } link: { samples: 0, unit: reqs } unlink: { samples: 0, unit: reqs } mkdir: { samples: 0, unit: reqs } rmdir: { samples: 0, unit: reqs } rename: { samples: 0, unit: reqs } getattr: { samples: 3, unit: reqs } setattr: { samples: 0, unit: reqs } getxattr: { samples: 0, unit: reqs } setxattr: { samples: 0, unit: reqs } statfs: { samples: 0, unit: reqs } sync: { samples: 0, unit: reqs } samedir_rename: { samples: 0, unit: reqs } crossdir_rename: { samples: 0, unit: reqs } - job_id: dd.0 snapshot_time: 1352085037 open: { samples: 1, unit: reqs } close: { samples: 1, unit: reqs } mknod: { samples: 0, unit: reqs } link: { samples: 0, unit: reqs } unlink: { samples: 0, unit: reqs } mkdir: { samples: 0, unit: reqs } rmdir: { samples: 0, unit: reqs } rename: { samples: 0, unit: reqs } getattr: { samples: 0, unit: reqs } setattr: { samples: 0, unit: reqs } getxattr: { samples: 0, unit: reqs } setxattr: { samples: 0, unit: reqs } statfs: { samples: 0, unit: reqs } sync: { samples: 2, unit: reqs } samedir_rename: { samples: 0, unit: reqs } crossdir_rename: { samples: 0, unit: reqs } </pre> </div></div> <p> The data operation stats is collected on OST, can one can check it via <tt>lctl get_param obdfilter.&#42;.job_stats</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl get_param obdfilter.lustre-OST0000.job_stats job_stats: - job_id: bash.0 snapshot_time: 1352085025 read: { samples: 0, unit: bytes, min: 0, max: 0, sum: 0 } write: { samples: 1, unit: bytes, min: 4, max: 4, sum: 4 } setattr: { samples: 0, unit: reqs } punch: { samples: 0, unit: reqs } sync: { samples: 0, unit: reqs } </pre> </div></div> <ul class="alternate" type="square"> <li>Clear job stats for specified job (or all jobs)</li> </ul> <p> One can clear the job stats for a certain MDT or OST by writing the proc file 'job_stats'.</p> <p> Clear stats for all job on testfs-OST0001:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl set_param obdfilter.testfs-OST0001.job_stats=clear </pre> </div></div> <p> Clear stats for job "dd.0" on lustre-MDT0000:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl set_param mdt.lustre-MDT0000.job_stats=dd.0 </pre> </div></div> <ul class="alternate" type="square"> <li>Configure cleanup interval</li> </ul> <p> By default, if some job doesn't have any activities for 600 seconds, it's stats will be cleared, this expiration value<br/> is tunable via mdt.<b>.job_cleanup_interval and obdfilter.</b>.job_cleanup_interval.</p> <p> for instance, change the cleanup interval to just over an hour (4000) seconds for MDT:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl conf_param lustre.mdt.job_cleanup_interval=4000 </pre> </div></div> <p> The 'job_cleanup_interval' can be set as 0 to disable the auto-cleanup. </p> <p><a href="http://review.whamcloud.com/1397" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1397</a></p> <p>follow-up patch which moves 'jobid_var' to global: <a href="http://review.whamcloud.com/1683" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1683</a></p> <p>Hello, Niu<br/> new patches for this feature will be landed for 2.2?</p> <p>Hi, Ihara</p> <p>The patch will not be landed for 2.2, which version it should be landed for is not decided yet.</p> <p>I have been advised that a filesystem name may not uniquely identify a lustre filesystem.</p> <p>I am not sure what a better choice for the command you have above is, but some thought as to an alternative to fs name would be valuable.</p> <blockquote> <p>I have been advised that a filesystem name may not uniquely identify a lustre filesystem.</p> <p>I am not sure what a better choice for the command you have above is, but some thought as to an alternative to fs name would be valuable.</p></blockquote> <p>Hi, Richard, fs name should be unique on a single MGS namespace, and most 'lctl conf_param' uses fsname to identify a filesystem. Do ou suggest that we'd set jobstats parameters per target server but not per fs? I'm not sure if I followed your comment correctly?</p> <p>The intent of the MGS was to provide config info for all the filesystems at a site, so the fs name is unique. If multiple MGS's are being used, on different nodes, the filesystem name could overlap &#8211; but you'd have to be masochistic to use the same filesystem name for two different filesystems at a single site. <br/> Masochistic to the point where I would say this should be disallowed by any and all configuration management systems, and if you do it by hand anyhow, you reap the unpleasant rewards.</p> <p>Nathan, it boggles my mind as well. But I know for a fact that folks out there have done it, because they complained about LMT not being able to handle two filesystem having exactly the same name. They seemed to think it was Livermore's responsibility to factor in additional information like IP addresses to uniquely identify filesystems with the same name. I of course declined.</p> <p>But one has to sympathize with the users. Configuring lustre is so horribly bad that something like the filesystem name is completely non-obvious to most people. You set it once in some cryptic way, and it none too clear from that point forward how it is used at all.</p> <p>Which I suppose is a long winded way of agreeing that filesystem names really need to be unique, and bending over backwards to differentiate filesystems with the same name is a path to madness. But we also need to promote the name to a first-class object that is used in a sane way in the command-line tools and throughout lustre. We also need to clearly document filesystem name usage.</p> <p>Thanks for the input. I agree that fs names are useful, for example:</p> <ul class="alternate" type="square"> <li>In the cases where there is only one MGS at a site, you need fs_name to distinguish the fs.</li> <li>In the cases where you have not mounted the fs, you can still identify the fs.</li> </ul> <p>So, how about supporting: <tt>&lt;mount point|fsname&gt;</tt>?</p> <p>If the mount point is not valid, then <tt>error</tt></p> <p>Just updated examples in this bug to be more clear, since it showed up in a Google search. I prefer not to use "lustre" as the fsname in examples, since it is very non-obvious that this needs to be replaced with the actual fsname and is not an fixed part of the parameter being specified (like the "sys.jobid_var" part is).</p> Related LU-2058 Development Rank 1|hzv4rr: Rank (Obsolete) 4306