https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15743 Lustre <p>It would be useful if "<tt>lfs find</tt>" could check/match xattrs on files.</p> <p>This should use "<tt>&#45;&#45;xattr &lt;name&gt;</tt>" to match files that have a <tt>&lt;name&gt;</tt> xattr, and "<tt>&#45;&#45;xattr-match &lt;name&gt;=&lt;value&gt;</tt>", where <tt>&lt;name&gt;</tt> and <tt>&lt;value&gt;</tt> are shell patterns that can match arbitrary text strings in the xattr. The output would be printed in plain text with escaped control characters.</p> LU-15743

"lfs find" is missing "-xattr" support

Improvement Minor Open Unresolved Thomas Bertschinger Andreas Dilger lad23dd lug23dd medium Thu, 14 Apr 2022 17:55:18 +0000 Mon, 5 Feb 2024 01:01:52 +0000 0 6 <p>Grabbing this one for developer day.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=defazio" class="user-hover" rel="defazio">defazio</a> did you ever make progress on this enhancement during developer day or afterward? With the closing of <a href="https://jira.whamcloud.com/browse/LU-16798" title="lfs find: new --jobid option" class="issue-link" data-issue-key="LU-16798"><del>LU-16798</del></a>, implementing the ability to do "<tt>lfs find --xattr user.job=REGEXP</tt>" would be particularly useful.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> Sorry, I never got around to doing this. Go ahead and assign it to someone else if you'd like, I'm not sure when I'll get to it.</p> <p>I've been looking into this and can submit a patch if this isn't already being worked on.<br/> One question is if this should have a printf directive?<br/> Right now if I wanted to print out the jobid of some files I would have to do</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lfs find /mnt/lustre --xattr user.job | xargs getfattr -m user.job -d </pre> </div></div> <p>The right way to do this with a printf directive isn't obvious to me, for example, would we want to only dump the user namespace (since e.g. "lustre.lov" wouldn't be helpful to dump)? Would we want to dump every user namespace xattr with newlines separating them, or in some other format? Would we want to dump only xattrs that match the supplied --xattr and --xattr-match arguments?<br/> With these questions I think just using <tt>xargs getfattr</tt> is the right way to go since then we don't have to make decisions for the user on how to print xattrs. But I wanted to ask about adding printf support in case it's important/desirable.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bertschinger" class="user-hover" rel="bertschinger">bertschinger</a>, it is a good question you raise, and I haven't really thought about that aspect very much. On the one hand, it would be useful to have a <tt>&#45;printf</tt> directive to allow printing specific xattrs if they are matched (e.g. sysadmin wants to know what is in "<tt>user.job</tt>" after finding files with a regexp). However, I don't think implementing that is a <b>requirement</b> for "<tt>lfs find &#45;&#45;xattr</tt>" to be implemented. As you wrote, this could be achieved with other tools after the fact, but that also adds overhead for every file accessed when "<tt>lfs</tt>" may already have this information in memory after checking the file.</p> <p>This should be filed as a separate improvement ticket, and we can discuss there the right syntax and options for printing the xattr. I suspect something like "<tt>getfattr</tt>" can do (dump in text, hex, base64, with options for which xattrs to print) but details TBD once we have some time to think about it. I don't think this is the most critical gap in the tool, but a nice to have.</p> <p>@Andreas thanks for the feedback. One other design / interface question...<br/> The description has --xattr and --xattr-match to separately test for the presence of an xattr and to match its contents.<br/> I think having two options is redundant because to test for the presence of an xattr without caring about its contents, a single <tt>&lt;name&gt;=&lt;value&gt;</tt> option can accept</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --xattr <span class="code-quote">"user.job="</span></pre> </div></div> <p>or even</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --xattr <span class="code-quote">"user.job"</span></pre> </div></div> <p>What do you think?<br/> This is problematic if you want to test for an xattr with '=' in its name, but that's likely an uncommon case, especially if the main use is searching JobIDs. If it's important though, it probably needs to allow escaping the '='.<br/> I did also consider the interface of other find implementations, but those that have xattr support seem to use --xattr to match the presence of any xattr at all (which isn't useful for Lustre since every file will match). So I don't think compatibility with other finds is worthwhile here but just wanted to bring it up for consideration.</p> <p>I think the following would be most useful:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-xattr name_pattern[=value_pattern] File has at least one extended attribute with name that matches shell pattern name_pattern, and value that matches shell pattern value_pattern if specified. </pre> </div></div> <p>While searching for <tt>user.job</tt> xattrs will be a common use case, I suspect that there will be other cases that need to be handled, so allowing a regexp for both the name and value is useful.</p> <p>"Thomas Bertschinger &lt;bertschinger@lanl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52804" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52804</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15743" title="&quot;lfs find&quot; is missing &quot;-xattr&quot; support" class="issue-link" data-issue-key="LU-15743">LU-15743</a> utils: add --xattr option to lfs find<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ab05f6780984bf74487cb284ec85bfcf31991d60</p> <p>There is a prototype patch for "GNU find" at <a href="https://gitlab.com/mweetman/findutils" class="external-link" target="_blank" rel="nofollow noopener">https://gitlab.com/mweetman/findutils</a> that adds "<tt>&#45;xattr</tt>" support. According to the <a href="https://gitlab.com/mweetman/findutils/-/commit/a4313fe967f7fffd3d3aabb54392ea4d902c44b2#c717da162f6446a03573da86a10d2da58fda9c94_978_995" class="external-link" target="_blank" rel="nofollow noopener">find.1 man page</a> in that patch, the "<tt>&#45;xattr PATTERN</tt>" option takes a regular expression for <tt>PATTERN</tt> by default, but it also allows the "<tt>&#45;regextype TYPE</tt>" option to change the regular expression type to one of <tt>emacs</tt>, <tt>posix-awk</tt>, <tt>posix-basic</tt>, <tt>posix-egrep</tt>, and <tt>posix-extended</tt>. I don't think there is any urgent need for this, since "<tt>lfs find</tt>" implements neither the "<tt>&#45;path</tt>" or "<tt>&#45;regex</tt>" tests.</p> <p>Andreas, adding a question here instead of on the Gerrit review about the regex matching issue, since it's more of a user interface / design question.</p> <blockquote><p>I'm not really thrilled that the regex is implicitly matching only part of the string for both the name and value, rather than explicitly requiring a wildcard for the remainder.  That makes it harder to match names/values that are subsets of each other, like "system.acl" "system.acl_default" or whatever.<br/> I would have assumed that it needs a regex "trusted.<b>" to explicitly match the name of all "trusted.</b>" xattrs.  Otherwise, it will be harder to match substrings in the xattr name.  I see that this is how getfattr works for "-m", but conversely "find -regex f" does not return all files with "f" in them (not for any regextype), only when "find -regex '.<b>f.</b>'" or "find -path '<b>f</b>'" is used does that happen.</p></blockquote> <p>It looks like GNU find uses a non-POSIX regex interface from Gnulib, and its matching function <tt>re_match</tt> returns the number of chars matched. So GNU find accomplishes this with basically</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (re_match(..., path, ...) == strlen(path)) ...</pre> </div></div> <p>It should be possible to achieve the same result with the POSIX regex interface using the pmatch[] argument to regexec, like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> regmatch_t pmatch[1]; regexec(re, input, 1, pmatch, 0); <span class="code-keyword">if</span> (pmatch[0].rm_so == 0 &amp;&amp; pmatch[0].rm_eo == strlen(input)) { <span class="code-comment">/* matched entire input string */</span> }</pre> </div></div> <p>Alternatively, with the current behavior, matching the entire string could be accomplished by the user with anchors: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lfs find --xattr "^user.*$ /mnt/lustre</pre> </div></div> <p>This could be documented/suggested in the man page to make it clear to the user if they want this behavior.</p> <p>A final option could be to use the Gnulib regex instead of POSIX regex for lfs find, but I'm not sure if that really makes sense.</p> <p>What do you think is the best option?</p> <p>I had suggested to use <tt>regex_comp(..., REG_NOSUB)</tt>, but I think the <tt>pmatch</tt> option is best. I think it is non-intuitive to make the user specify '<tt>^</tt>' or '<tt>$</tt>' to terminate the regexp. </p> Related LU-5170 LU-13031 LU-16798 LU-15837 LU-17219 LU-16760 Development Rank 1|i02n3j: Rank (Obsolete) 9223372036854775807