Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-15743

"lfs find" is missing "-xattr" support

Details

    • 9223372036854775807

    Description

      It would be useful if "lfs find" could check/match xattrs on files.

      This should use "--xattr <name>" to match files that have a <name> xattr, and "--xattr-match <name>=<value>", where <name> and <value> are shell patterns that can match arbitrary text strings in the xattr. The output would be printed in plain text with escaped control characters.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-17219 lfs find: add ability to print extended attributes

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-18889 "lfs find -printf" optimization

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-15837 "lfs find -printf" improvements

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-16760 "lfs find" support for fscrypt and other file attributes

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-5170 lfs usability

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-13031 store JobID of program that created file in inodes at create time

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-16798 lfs find: new --jobid option

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            [LU-15743] "lfs find" is missing "-xattr" support
            gerrit Gerrit Updater added a comment -

            "Thomas Bertschinger <bertschinger@lanl.gov>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52804
            Subject: LU-15743 utils: add --xattr option to lfs find
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ab05f6780984bf74487cb284ec85bfcf31991d60

            gerrit Gerrit Updater added a comment - "Thomas Bertschinger <bertschinger@lanl.gov>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52804 Subject: LU-15743 utils: add --xattr option to lfs find Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ab05f6780984bf74487cb284ec85bfcf31991d60
            adilger Andreas Dilger added a comment -

            I think the following would be most useful:

            -xattr name_pattern[=value_pattern]
              File has at least one extended attribute with name that
              matches shell pattern name_pattern, and value that matches
              shell pattern value_pattern if specified.

            While searching for user.job xattrs will be a common use case, I suspect that there will be other cases that need to be handled, so allowing a regexp for both the name and value is useful.

            adilger Andreas Dilger added a comment - I think the following would be most useful: -xattr name_pattern[=value_pattern] File has at least one extended attribute with name that matches shell pattern name_pattern, and value that matches shell pattern value_pattern if specified. While searching for user.job xattrs will be a common use case, I suspect that there will be other cases that need to be handled, so allowing a regexp for both the name and value is useful.
            bertschinger Thomas Bertschinger added a comment -

            @Andreas thanks for the feedback. One other design / interface question...
            The description has --xattr and --xattr-match to separately test for the presence of an xattr and to match its contents.
            I think having two options is redundant because to test for the presence of an xattr without caring about its contents, a single <name>=<value> option can accept

            --xattr "user.job="

            or even

            --xattr "user.job"

            What do you think?
            This is problematic if you want to test for an xattr with '=' in its name, but that's likely an uncommon case, especially if the main use is searching JobIDs. If it's important though, it probably needs to allow escaping the '='.
            I did also consider the interface of other find implementations, but those that have xattr support seem to use --xattr to match the presence of any xattr at all (which isn't useful for Lustre since every file will match). So I don't think compatibility with other finds is worthwhile here but just wanted to bring it up for consideration.

            bertschinger Thomas Bertschinger added a comment - @Andreas thanks for the feedback. One other design / interface question... The description has --xattr and --xattr-match to separately test for the presence of an xattr and to match its contents. I think having two options is redundant because to test for the presence of an xattr without caring about its contents, a single <name>=<value> option can accept --xattr "user.job=" or even --xattr "user.job" What do you think? This is problematic if you want to test for an xattr with '=' in its name, but that's likely an uncommon case, especially if the main use is searching JobIDs. If it's important though, it probably needs to allow escaping the '='. I did also consider the interface of other find implementations, but those that have xattr support seem to use --xattr to match the presence of any xattr at all (which isn't useful for Lustre since every file will match). So I don't think compatibility with other finds is worthwhile here but just wanted to bring it up for consideration.
            adilger Andreas Dilger added a comment -

            bertschinger, it is a good question you raise, and I haven't really thought about that aspect very much. On the one hand, it would be useful to have a -printf directive to allow printing specific xattrs if they are matched (e.g. sysadmin wants to know what is in "user.job" after finding files with a regexp). However, I don't think implementing that is a requirement for "lfs find --xattr" to be implemented. As you wrote, this could be achieved with other tools after the fact, but that also adds overhead for every file accessed when "lfs" may already have this information in memory after checking the file.

            This should be filed as a separate improvement ticket, and we can discuss there the right syntax and options for printing the xattr. I suspect something like "getfattr" can do (dump in text, hex, base64, with options for which xattrs to print) but details TBD once we have some time to think about it. I don't think this is the most critical gap in the tool, but a nice to have.

            adilger Andreas Dilger added a comment - bertschinger , it is a good question you raise, and I haven't really thought about that aspect very much. On the one hand, it would be useful to have a -printf directive to allow printing specific xattrs if they are matched (e.g. sysadmin wants to know what is in " user.job " after finding files with a regexp). However, I don't think implementing that is a requirement for " lfs find --xattr " to be implemented. As you wrote, this could be achieved with other tools after the fact, but that also adds overhead for every file accessed when " lfs " may already have this information in memory after checking the file. This should be filed as a separate improvement ticket, and we can discuss there the right syntax and options for printing the xattr. I suspect something like " getfattr " can do (dump in text, hex, base64, with options for which xattrs to print) but details TBD once we have some time to think about it. I don't think this is the most critical gap in the tool, but a nice to have.
            bertschinger Thomas Bertschinger added a comment -

            I've been looking into this and can submit a patch if this isn't already being worked on.
            One question is if this should have a printf directive?
            Right now if I wanted to print out the jobid of some files I would have to do

            lfs find /mnt/lustre --xattr user.job | xargs getfattr -m user.job -d

            The right way to do this with a printf directive isn't obvious to me, for example, would we want to only dump the user namespace (since e.g. "lustre.lov" wouldn't be helpful to dump)? Would we want to dump every user namespace xattr with newlines separating them, or in some other format? Would we want to dump only xattrs that match the supplied --xattr and --xattr-match arguments?
            With these questions I think just using xargs getfattr is the right way to go since then we don't have to make decisions for the user on how to print xattrs. But I wanted to ask about adding printf support in case it's important/desirable.

            bertschinger Thomas Bertschinger added a comment - I've been looking into this and can submit a patch if this isn't already being worked on. One question is if this should have a printf directive? Right now if I wanted to print out the jobid of some files I would have to do lfs find /mnt/lustre --xattr user.job | xargs getfattr -m user.job -d The right way to do this with a printf directive isn't obvious to me, for example, would we want to only dump the user namespace (since e.g. "lustre.lov" wouldn't be helpful to dump)? Would we want to dump every user namespace xattr with newlines separating them, or in some other format? Would we want to dump only xattrs that match the supplied --xattr and --xattr-match arguments? With these questions I think just using xargs getfattr is the right way to go since then we don't have to make decisions for the user on how to print xattrs. But I wanted to ask about adding printf support in case it's important/desirable.
            defazio Gian-Carlo Defazio added a comment -

            adilger Sorry, I never got around to doing this. Go ahead and assign it to someone else if you'd like, I'm not sure when I'll get to it.

            defazio Gian-Carlo Defazio added a comment - adilger Sorry, I never got around to doing this. Go ahead and assign it to someone else if you'd like, I'm not sure when I'll get to it.
            adilger Andreas Dilger added a comment -

            defazio did you ever make progress on this enhancement during developer day or afterward? With the closing of LU-16798, implementing the ability to do "lfs find --xattr user.job=REGEXP" would be particularly useful.

            adilger Andreas Dilger added a comment - defazio did you ever make progress on this enhancement during developer day or afterward? With the closing of LU-16798 , implementing the ability to do " lfs find --xattr user.job=REGEXP " would be particularly useful.
            defazio Gian-Carlo Defazio added a comment -

            Grabbing this one for developer day.

            defazio Gian-Carlo Defazio added a comment - Grabbing this one for developer day.

            People

              bertschinger Thomas Bertschinger
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: