Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-15743

"lfs find" is missing "-xattr" support

Details

    • 9223372036854775807

    Description

      It would be useful if "lfs find" could check/match xattrs on files.

      This should use "--xattr <name>" to match files that have a <name> xattr, and "--xattr-match <name>=<value>", where <name> and <value> are shell patterns that can match arbitrary text strings in the xattr. The output would be printed in plain text with escaped control characters.

      Attachments

        Issue Links

          Activity

            [LU-15743] "lfs find" is missing "-xattr" support

            "Thomas Bertschinger <bertschinger@lanl.gov>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52804
            Subject: LU-15743 utils: add --xattr option to lfs find
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ab05f6780984bf74487cb284ec85bfcf31991d60

            gerrit Gerrit Updater added a comment - "Thomas Bertschinger <bertschinger@lanl.gov>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52804 Subject: LU-15743 utils: add --xattr option to lfs find Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ab05f6780984bf74487cb284ec85bfcf31991d60

            I think the following would be most useful:

            -xattr name_pattern[=value_pattern]
                          File has at least one extended attribute with name that
                          matches shell pattern name_pattern, and value that matches
                          shell pattern value_pattern if specified.
            

            While searching for user.job xattrs will be a common use case, I suspect that there will be other cases that need to be handled, so allowing a regexp for both the name and value is useful.

            adilger Andreas Dilger added a comment - I think the following would be most useful: -xattr name_pattern[=value_pattern] File has at least one extended attribute with name that matches shell pattern name_pattern, and value that matches shell pattern value_pattern if specified. While searching for user.job xattrs will be a common use case, I suspect that there will be other cases that need to be handled, so allowing a regexp for both the name and value is useful.

            @Andreas thanks for the feedback. One other design / interface question...
            The description has --xattr and --xattr-match to separately test for the presence of an xattr and to match its contents.
            I think having two options is redundant because to test for the presence of an xattr without caring about its contents, a single <name>=<value> option can accept

            --xattr "user.job="

            or even

            --xattr "user.job"

            What do you think?
            This is problematic if you want to test for an xattr with '=' in its name, but that's likely an uncommon case, especially if the main use is searching JobIDs. If it's important though, it probably needs to allow escaping the '='.
            I did also consider the interface of other find implementations, but those that have xattr support seem to use --xattr to match the presence of any xattr at all (which isn't useful for Lustre since every file will match). So I don't think compatibility with other finds is worthwhile here but just wanted to bring it up for consideration.

            bertschinger Thomas Bertschinger added a comment - @Andreas thanks for the feedback. One other design / interface question... The description has --xattr and --xattr-match to separately test for the presence of an xattr and to match its contents. I think having two options is redundant because to test for the presence of an xattr without caring about its contents, a single <name>=<value> option can accept --xattr "user.job=" or even --xattr "user.job" What do you think? This is problematic if you want to test for an xattr with '=' in its name, but that's likely an uncommon case, especially if the main use is searching JobIDs. If it's important though, it probably needs to allow escaping the '='. I did also consider the interface of other find implementations, but those that have xattr support seem to use --xattr to match the presence of any xattr at all (which isn't useful for Lustre since every file will match). So I don't think compatibility with other finds is worthwhile here but just wanted to bring it up for consideration.

            bertschinger, it is a good question you raise, and I haven't really thought about that aspect very much. On the one hand, it would be useful to have a -printf directive to allow printing specific xattrs if they are matched (e.g. sysadmin wants to know what is in "user.job" after finding files with a regexp). However, I don't think implementing that is a requirement for "lfs find --xattr" to be implemented. As you wrote, this could be achieved with other tools after the fact, but that also adds overhead for every file accessed when "lfs" may already have this information in memory after checking the file.

            This should be filed as a separate improvement ticket, and we can discuss there the right syntax and options for printing the xattr. I suspect something like "getfattr" can do (dump in text, hex, base64, with options for which xattrs to print) but details TBD once we have some time to think about it. I don't think this is the most critical gap in the tool, but a nice to have.

            adilger Andreas Dilger added a comment - bertschinger , it is a good question you raise, and I haven't really thought about that aspect very much. On the one hand, it would be useful to have a -printf directive to allow printing specific xattrs if they are matched (e.g. sysadmin wants to know what is in " user.job " after finding files with a regexp). However, I don't think implementing that is a requirement for " lfs find --xattr " to be implemented. As you wrote, this could be achieved with other tools after the fact, but that also adds overhead for every file accessed when " lfs " may already have this information in memory after checking the file. This should be filed as a separate improvement ticket, and we can discuss there the right syntax and options for printing the xattr. I suspect something like " getfattr " can do (dump in text, hex, base64, with options for which xattrs to print) but details TBD once we have some time to think about it. I don't think this is the most critical gap in the tool, but a nice to have.

            I've been looking into this and can submit a patch if this isn't already being worked on.
            One question is if this should have a printf directive?
            Right now if I wanted to print out the jobid of some files I would have to do

            lfs find /mnt/lustre --xattr user.job | xargs getfattr -m user.job -d 

            The right way to do this with a printf directive isn't obvious to me, for example, would we want to only dump the user namespace (since e.g. "lustre.lov" wouldn't be helpful to dump)? Would we want to dump every user namespace xattr with newlines separating them, or in some other format? Would we want to dump only xattrs that match the supplied --xattr and --xattr-match arguments?
            With these questions I think just using xargs getfattr is the right way to go since then we don't have to make decisions for the user on how to print xattrs. But I wanted to ask about adding printf support in case it's important/desirable.

            bertschinger Thomas Bertschinger added a comment - I've been looking into this and can submit a patch if this isn't already being worked on. One question is if this should have a printf directive? Right now if I wanted to print out the jobid of some files I would have to do lfs find /mnt/lustre --xattr user.job | xargs getfattr -m user.job -d The right way to do this with a printf directive isn't obvious to me, for example, would we want to only dump the user namespace (since e.g. "lustre.lov" wouldn't be helpful to dump)? Would we want to dump every user namespace xattr with newlines separating them, or in some other format? Would we want to dump only xattrs that match the supplied --xattr and --xattr-match arguments? With these questions I think just using xargs getfattr is the right way to go since then we don't have to make decisions for the user on how to print xattrs. But I wanted to ask about adding printf support in case it's important/desirable.

            adilger Sorry, I never got around to doing this. Go ahead and assign it to someone else if you'd like, I'm not sure when I'll get to it.

            defazio Gian-Carlo Defazio added a comment - adilger Sorry, I never got around to doing this. Go ahead and assign it to someone else if you'd like, I'm not sure when I'll get to it.

            defazio did you ever make progress on this enhancement during developer day or afterward? With the closing of LU-16798, implementing the ability to do "lfs find --xattr user.job=REGEXP" would be particularly useful.

            adilger Andreas Dilger added a comment - defazio did you ever make progress on this enhancement during developer day or afterward? With the closing of LU-16798 , implementing the ability to do " lfs find --xattr user.job=REGEXP " would be particularly useful.

            Grabbing this one for developer day.

            defazio Gian-Carlo Defazio added a comment - Grabbing this one for developer day.

            People

              bertschinger Thomas Bertschinger
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: