https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1996 Lustre <p>We propose a patch (gerrit will follow) to add a jobid field to the changelogs for job activity tracking. Unlike jobstats, changelog allow fine grained tracking as they contain the FIDs and the events.</p> <p>Note that this patch introduces a new jobid field (32chars long) to the changelog record structures.</p> LU-1996

Fine-grained job activity tracking using changelogs

New Feature Minor Resolved Fixed Niu Yawei Henri Doreau patch Thu, 20 Sep 2012 09:23:34 +0000 Wed, 15 Dec 2021 08:57:59 +0000 Sat, 22 Nov 2014 00:27:51 +0000 Lustre 2.4.0 Lustre 2.7.0 0 11 <p>Patch is at: <a href="http://review.whamcloud.com/#change,4060" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4060</a></p> <p>Jodi please can someone review this as part of the HSM efforts</p> <p>I didn't mention it but actually this patch is not related to the HSM project.</p> <p>This seems to be related to the JobStats work that you did. Would you be able to take this one as well?</p> <p>I got no feedback on this patch so far.<br/> Any update on this?</p> <p>Presumably this patch would break compatibility with the existing ChangeLog on-disk and in-memory record format. Is there any attempt made at interoperability for old applications (e.g. some kind of flag on open to indicate if the new or old changelog structure is in use)? Looking at the patch appears that there is only one-way compatibility (i.e. new clients/apps with old records).</p> <p>Right, we did not focus on the second case. I'll update the patch with what you proposed on gerrit: opening the changelog w/ or w/o CHANGELOG_FLAG_JOBID determines whether the records get delivered w/ the jobid field.</p> <p>What is the LLAPI compatibility policy? Shall I preserve binary compatibility? If so, I'd store jobid in cr_name (the variable length field), even though cr_name already contains name and sname...</p> <p>What do you think?</p> <p>I'd rather not hack around with the changelog_ext_rec structure, which we'll have to carry around for a long time in the future. It should be possible to introduce a new changelog_rec_v2 or similar, and possibly a new CHANGELOG_REC_V2 type to distinguish it. We don't have to keep compatibility for the old record type forever, since Changelogs are relatively infrequently used, and it is possible to start printing a warning with 2.5 when opening the changelog without CHANGELOG_FLAG_JOBID, but we need to keep the compatibility around for at least a few releases.</p> <p>Henri,<br/> Is there likely to be any further movement on this issue?<br/> If not, may I mark it as resolved?<br/> Thanks,<br/> ~ jfc.</p> <p>Henri is still working on this feature<br/> and we still want to have it in Lustre.<br/> Please keep it opened.<br/> Thanks,</p> <ul class="alternate" type="square"> <li>Thomas</li> </ul> <p>FYI, I believe I have found a bug introduced by this work. I am tracking it at <a href="https://jira.whamcloud.com/browse/LU-5862" title="Jobstats tracking in changelogs doesn&#39;t support jobid_var=disable" class="issue-link" data-issue-key="LU-5862"><del>LU-5862</del></a>. I haven't dug into the cause yet, but when jobid_var is set to 'disable' (the default), the changelog records emitted by "lfs changelog" have the filename in the jobid field.</p> <p>I'm closing this bug, since the feature patch to enable the jobid in ChangeLogs has landed. There is a bug in the patch that is being tracked via <a href="https://jira.whamcloud.com/browse/LU-5862" title="Jobstats tracking in changelogs doesn&#39;t support jobid_var=disable" class="issue-link" data-issue-key="LU-5862"><del>LU-5862</del></a>.</p> Related LU-5862 LU-5631 LU-15372 Development Rank 1|hzvv87: Rank (Obsolete) 9063