https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13031 Lustre <p>It would be useful to store the JobID that created a file in an xattr on the MDT inode. This would allow tracking the source of files after the fact, and can help users with the provenance of their file data (e.g. which run the files came from, which can then be used to determine runtime parameters used, whether the run was successful or not, etc.).</p> <p>It isn't clear whether this would be needed on the OST inodes as well, but I think initially the MDT inode would be enough, as this would be the only xattr visible to users. The OST xattr would only be accessible for low-level scanning tools.</p> LU-13031

store JobID of program that created file in inodes at create time

Improvement Minor Resolved Fixed Thomas Bertschinger Andreas Dilger LTS15 llnl lug23dd medium Thu, 28 Nov 2019 18:56:07 +0000 Thu, 7 Dec 2023 16:47:25 +0000 Mon, 7 Aug 2023 14:06:41 +0000 Lustre 2.16.0 Lustre 2.15.4 0 12 <p>I agree this would be useful. If you're not in a hurry, we would like to try to implement it.</p> <p>Olaf,<br/> I don't think anyone is working on this right now, so I'd be happy for tour contribution in this area. The jobid is large enough (32 bytes) that it is reasonable to store it in its own xattr on the file. For ext4's sake, the xattr should be named "job" (3-byte xattr names are the most efficient to store), and probably stored in the "lustre" or "user" namespace so that it is visible to regular users, as it is probably not critical if users can change or remove this xattr for whatever reason.</p> <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50982" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50982</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> jobstats: store jobid in xattr when files are created<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 26c63b60e0dcd145a8d032a9f4780c921435e184</p> <p>@Patrick Farrell<br/> Here's a link to my question on the mailing list archive <a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2023-May/018629.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2023-May/018629.html</a> If you want to respond with your comments either in the mailing list or in the ticket here to continue the discussion.</p> <p>Thanks, Thomas.</p> <p>Could you lay out the implications of using user vs lustre namespaces as you see them?  What do we gain, what we do lose, and/or what requirements can't we meet in certain settings?</p> <p>Requirements (or at least, Wants):</p> <ul> <li>Can preserve xattr with tools like `cp` and `rsync` (it's not useful if you copy your data and the jobid gets overwritten with stuff like "cp.1234")</li> <li>Can preserve xattr when moving files to other filesystems, e.g., to an archive, since the jobid is likely useful to keep around long term</li> </ul> <p><b>User</b> namespace<br/> Pros:</p> <ul class="alternate" type="square"> <li>Enables things like `cp --preserve=xattr` and `rsync --xattrs`</li> <li>Allows interoperability with other filesystems (xattr can be preserved if a file is copied to an FS that doesn't know about the "lustre" namespace)</li> </ul> <p>Cons:</p> <ul class="alternate" type="square"> <li>violates expectation that the kernel will not write user data, which could be problematic if this becomes enforced at some point</li> </ul> <p><b>Lustre</b> namespace:<br/> Pros:</p> <ul class="alternate" type="square"> <li>Allows flexibility in terms of being written from both user and kernel space; we can decide our own rules</li> </ul> <p>Cons:</p> <ul class="alternate" type="square"> <li>Lustre xattr is "virtual" and must be backed by an xattr in a traditional namespace (example, lustre.lov and trusted.lov) since the kernel (AFAICT) will reject storing a lustre.* xattr directly, which increases complexity.</li> <li>Prevents interoperability-- standard tools like `rsync` won't be able to preserve this xattr because tools that copy files to other filesystems will have to be specially coded to translate lustre.job into user.job</li> </ul> <p><b>Trusted</b> namespace: I think this would only be considered as backing the lustre namespace and wouldn't be used on its own.</p> <p><b>System</b> namespace:<br/> Pros: <br/> -Arguably the most correct choice. According to man 7 xattr: "Extended system attributes are used by the kernel to store system objects...  Read and write access permissions to system attributes depend on the policy ... implemented by filesystems in the kernel." So this fits our need of being able to write it from both user and kernel processes.<br/> Cons: <br/> -Is not interoperable with other filesystems because other filesystems likely will not allow user processes to write a "system.job" xattr.</p> <p> </p> <p>My opinion is that the user namespace is the best choice because every other choice fails to meet the goals above, but I will admit that not being very experienced with the kernel side of things, I can't say how severe the problem with the user namespace is.</p> <p>Thank you for that; I'm persuaded we should do the user xattr, the issues around interop and user tools look decisive.  If we're forced to change tack in the future, we can deal with that then.  (I think I was the only one actively objecting, Andreas indicated OK at LUG, so you should be good to go.)</p> <p>I'm inclined toward the <tt>user.job</tt> xattr, but it is never a good idea to unilaterally make policy decisions in the kernel that cannot be changed.</p> <p>As such, it might make sense to have a tunable parameter like "<tt>mdt.&#42;.job_xattr=user.job</tt>" (or <tt>mdd.&#42;.job_xattr</tt> is possibly better, though e.g. "<tt>mdt.&#42;.job_cleanup_interval</tt>" already exists), and then this could be changed in the future if there is some conflict (e.g. some site already uses the "<tt>user.job</tt>" xattr for some other purpose).</p> <p>I don't think the <tt>job_xattr</tt> should allow totally arbitrary values (e.g. overwriting <tt>trusted.lov</tt> or <tt>trusted.lma</tt> or <tt>security.&#42;</tt> would be bad). One option is to only allow a limited selection of valid xattr namespaces, and possibly names:</p> <ul class="alternate" type="square"> <li><tt>NONE</tt> to turn this feature off</li> <li><tt>user</tt>, or <tt>trusted</tt> or <tt>system</tt> (if admin wants to restrict the ability of regular users to change this value?), with implicit "<tt>.job</tt>" added</li> <li><tt>user.&#42;</tt> (or <tt>trusted.&#42;</tt> or <tt>system.&#42;</tt>) to also allow specifying the xattr name</li> </ul> <p>If we allow the xattr name portion to be specified (which I'm not sure about, but putting it out for completeness), it should have some reasonable limits:</p> <ul class="alternate" type="square"> <li>&lt;= 7 characters long to avoid wasting valuable xattr space in the inode</li> <li>only ASCII alpha-numeric characters, plus "<tt>.</tt>" for the name separator, to avoid spaces, control characters, symbols (<tt>=</tt>, <tt>,</tt>, etc.) that might cause problems</li> <li>should not conflict with other known xattrs, which is tricky if we allow the name to be arbitrary. Possibly if in <tt>trusted</tt> (and <tt>system</tt>?) it should only allow <tt>trusted.job</tt> to avoid future conflicts?</li> <li>maybe restrict it to contain "<tt>job</tt>" (or maybe "<tt>pbs</tt>", "<tt>slurm</tt>", ...) to reduce the chance of namespace clashes in <tt>user</tt> or <tt>system</tt>? However, I'm reluctant to restrict names in <tt>user</tt> since this <em>shouldn't</em> have any fatal side effects (e.g. data corruption like in <tt>trusted</tt> or <tt>system</tt>), and the admin is supposed to know what they are doing...</li> </ul> <p>Thoughts?</p> <p>For the implementation, any such restrictions/checking would be done in the sysfs write handler when the xattr is first set, and for actual xattr creation this would just copy the job_xattr string during actual file creation with no need for additional sanity checking, since it should always be consistent.</p> <p>For the "don't want <tt>user.job</tt> to be <tt>cp.12345</tt>" issue, I <em>think</em> that should be handled properly. Initially, the xattr <em>would</em> be set to <tt>cp.PID</tt> or <tt>SLURM_JOBID</tt> or whatever by the MDS internally at file creation time. However, if <tt>cp</tt>, <tt>rsync</tt>, <tt>tar</tt>, <tt>dcp</tt>, <tt>dsync</tt> ... is also copying xattrs, the initial value would be overwritten later in the copy process, which is what we would want I think.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50982/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50982/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> jobstats: store jobid in xattr when files are created<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 23a2db28dcf1422a6a6da575e907fd257106d402</p> <p>Landed for 2.16</p> <p>The new maloo test fail in interop.</p> <p>Does interop refer to some testing that mixes different client and server versions? Is the failure occurring when the server version is older than the client and the MDS doesn't have this new change?</p> <p>If it just needs a version check like:</p> <p><tt>(( $MDS1_VERSION &gt;= $(version_code 2.15.??) )) || skip ...</tt></p> <p>that should be simple enough, I can submit a fix for that.</p> <p>The version I get when I check out this commit and run `lctl get_param version` is 2.15.57. Is that the correct code to put in the version check? Or do I need one higher than that (since I guess you could check out a commit newer than the 2.15.57 tag but older than this change and its version would pass...). If I need one higher than 2.15.57, do I use 2.15.58, 2.16.0, or something else?</p> <p>Yes, you've got this exactly right.  That's what interop is and that check is what's needed.</p> <p>Generally you'd just use 2.15.57, the tag that was live when the patch was merged.  We don't do interop testing between different development versions, just between dev versions and older full versions.  It's also fine if in some narrow scenario we skip a test that would pass.</p> <p>So that MDS version check you suggested for 2.15.57 is what's needed.  Oops on the reviewers (ie, including me) for not noting that at the time.</p> <p>The <tt>version_code</tt> helper supports 4-component version numbers, so since this is being done after the landing it is helpful to include the full version from "<tt>git describe</tt>" of the commit hash of the patch itself. If it were before landing it could use the 4-component version of the patch before landing, even though it is slightly inaccurate.. </p> <p>It is also helpful to include a brief description of why the test is being skipped. Please see some examples in patch <a href="https://review.whamcloud.com/52009" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52009</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16097" title="Mitigate the affect from pre-allocated quota after the quota is over limit" class="issue-link" data-issue-key="LU-16097"><del>LU-16097</del></a> tests: skip quota subtests in interop</tt>". </p> <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52095" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52095</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> tests: skip sanity/test_205h,205i in interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1532029b43cfdc41b6747ea2d4076f12d28e8be0</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52095/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52095/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> tests: skip sanity/test_205h,205i in interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a6df532162556028c2ab9b974989fc0cca68d4fe</p> <p>"Xing Huang &lt;hxing@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52332" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52332</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> tests: avoid new user xattr in interop<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: ed56abd1070e9812e55728b14c760580b03074e0</p> <p>Thomas, I just realized that this is storing the xattr on the MDT inode, but it would also be useful if the same xattr was also stored on the OST objects for debugging/scanning/recovery purposes. Is this something that you would be interested to work on?</p> <p>Hi Andreas,</p> <p>I'd be willing to work on adding that in. I'm pretty busy at the moment so I likely won't be able to start working on it for 2-3 weeks, but if it can wait that long I'm happy to give it a shot.</p> <p>No rush. For some reason I thought this was part of the original implementation, but upon looking at the patch again I saw it was only on the MDT inode.</p> <p>I think it makes sense to put this in <tt>ofd_attr_handle_id()</tt> where it is initializing the object ownership the first time the OST object is written to (in the "<tt>if (mask != 0)</tt>" block if it cleared the <tt>SUID</tt>/<tt>SGID</tt> bits). That allows a "one shot" setting of the <tt>user.job</tt> xattr without having to check each time whether it exists or not. The parameter <tt>obdfilter.&#42;.jobid_xattr</tt> should be used like <tt>mdt.&#42;.jobid_xattr</tt> to control the xattr name.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52332/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52332/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> tests: avoid new user xattr in interop<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: fac50dcc27f09384981d995e1234943f06a17969</p> <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53367" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53367</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13031" title="store JobID of program that created file in inodes at create time" class="issue-link" data-issue-key="LU-13031"><del>LU-13031</del></a> ofd: add jobid xattr to ost object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3bfb3d3739574bf298fb875257dcaab8bc77a61</p> Duplicate LU-11901 Gantt End to Start LU-16798 Related LU-16007 LUDOC-373 LU-15743 LU-17219 Development Rank 1|i00q67: Rank (Obsolete) 9223372036854775807