Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-17316

make more changelog metadata visible on clients

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • Lustre 2.15.3
    • 3
    • 9223372036854775807

    Description

      For an application running on a lustre client consuming changelogs, I believe the following would be useful. Both of these are available on the MDS via lctl get_param but there is currently no way (that I'm aware of) to get this information on a client node.

      On Lustre servers, we avoid running software other than critical management and monitoring tools, for both security and reliability reasons. I believe that is common practice. That means that something like a policy engine should be able to get whatever changelog information it needs on a lustre client (not server) node. Our local concern at the moment is Starfish, but I think below would be equally useful for others.

      1. The ability to obtain the list of registered changelog users for a given MDT.

      Currently, the application needs the user to specify which changelog user (e.g. "cl2") to use. For cases where the user is only running one application which consumes changelogs, and that application interacts with only one user, this isn't necessary (I suspect this is most cases, but I may be wrong).

      In addition, the application currently can't confirm a changelog user is valid. One example we've seen is that it would be useful to be able to detect that the changelog user was deregistered while the application was using it.

      2. The ability to learn the next available changelog record ID and the last changelog record ID for a given MDT.

      The application could adapt to the number of outstanding changelog records by creating more consumer threads when then are many changelogs available, and reducing thread count when there are few. The application could also better handle errors, such as when changelogs are cleared by a sysadmin to free space while the application is running.

      It's possible to obtain the next changelog record if there is one which hasn't yet been cleared, with "lfs changelog | head -n1" or similar. But if there is no changelog record yet to be cleared, that technique doesn't work.

      Attachments

        Issue Links

          Activity

            [LU-17316] make more changelog metadata visible on clients
            ofaaland Olaf Faaland added a comment -

            Andreas wrote:

            > Mike, what I did see is that "lctl --device changelog_deregister -u test" works to deregister this user, but only the username doesn't work on the client:
            >
            > # lfs changelog_clear testfs-MDT0000 test 1
            > lfs changelog_clear: cannot purge records for 'test': Invalid argument (22)
            > # lfs changelog_clear testfs-MDT0000 cl2-test 1
            > #
            > This should probably be usable with just the username?

            The reason that failed is the MDC only knows about numeric changelog user ids. llapi_changelog_clear() takes an argument of the form "cl#-name" or "cl#" and in the former case throws away "-name", passing only "cl#" to the kernel. mdc_changelog.c:chlg_write() then parses the input with this code:

                    if (sscanf(kbuf, "clear:cl%u:%llu", &reader, &record) == 2)
                            rc = chlg_clear(crs, reader, record);
                    else
                            rc = -EINVAL;
            

            note that it's expecting the changelog user to be specified as "cl#".

            So it's not a bug, per se, more that changelog user names aren't fully implemented. I'll look into what it would take to allow changelog user names to be used anywhere cl# forms are already accepted.

            ofaaland Olaf Faaland added a comment - Andreas wrote: > Mike, what I did see is that "lctl --device changelog_deregister -u test" works to deregister this user, but only the username doesn't work on the client: > > # lfs changelog_clear testfs-MDT0000 test 1 > lfs changelog_clear: cannot purge records for 'test': Invalid argument (22) > # lfs changelog_clear testfs-MDT0000 cl2-test 1 > # > This should probably be usable with just the username? The reason that failed is the MDC only knows about numeric changelog user ids. llapi_changelog_clear() takes an argument of the form "cl#-name" or "cl#" and in the former case throws away "-name", passing only "cl#" to the kernel. mdc_changelog.c:chlg_write() then parses the input with this code: if (sscanf(kbuf, "clear:cl%u:%llu", &reader, &record) == 2) rc = chlg_clear(crs, reader, record); else rc = -EINVAL; note that it's expecting the changelog user to be specified as "cl#". So it's not a bug, per se, more that changelog user names aren't fully implemented. I'll look into what it would take to allow changelog user names to be used anywhere cl# forms are already accepted.

            To clarify, we don't run things like Starfish or Robinhood on the lustre server nodes themselves, for security, performance, and reliability reasons (possibly we're making the wrong choices, but that's the line of reasoning). So exporting more changelog information to the client potentially allows such services to run better.

            Also, I'm not asking Whamcloud do this work. I'm just trying to make sure the proposal is sound, and to get a sense of how experienced a developer would need to be to have any hope of getting it done. Low granularity is fine for my purposes.

            ofaaland Olaf Faaland added a comment - To clarify, we don't run things like Starfish or Robinhood on the lustre server nodes themselves, for security, performance, and reliability reasons (possibly we're making the wrong choices, but that's the line of reasoning). So exporting more changelog information to the client potentially allows such services to run better. Also, I'm not asking Whamcloud do this work. I'm just trying to make sure the proposal is sound, and to get a sense of how experienced a developer would need to be to have any hope of getting it done. Low granularity is fine for my purposes.
            ofaaland Olaf Faaland added a comment -

            Andreas,

            > Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"?

            Yes, that would help.

            ofaaland Olaf Faaland added a comment - Andreas, > Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"? Yes, that would help.

            Olaf, there was also LU-17290 filed that requested a "more graceful" handling of idle changelog users when they need to be evicted because of too many unprocessed records.

            Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"?

            I'm not against your proposal to export more changelog information to the client, but also want to ensure that these requests are implemented in a way that is aligned.

            adilger Andreas Dilger added a comment - Olaf, there was also LU-17290 filed that requested a "more graceful" handling of idle changelog users when they need to be evicted because of too many unprocessed records. Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"? I'm not against your proposal to export more changelog information to the client, but also want to ensure that these requests are implemented in a way that is aligned.
            ofaaland Olaf Faaland added a comment -

            Andreas,
            Thanks, I'd forgotten that users could name changelog users. I agree, that helps. It would still be useful to have a mechanism to confirm that a changelog user (user-named or not) is valid, and allowing 'lfs' to report the list would be one way of doing that.

            ofaaland Olaf Faaland added a comment - Andreas, Thanks, I'd forgotten that users could name changelog users. I agree, that helps. It would still be useful to have a mechanism to confirm that a changelog user (user-named or not) is valid, and allowing 'lfs' to report the list would be one way of doing that.
            ofaaland Olaf Faaland added a comment -

            Hi Mike,
            Yes, this is about adding functionality to 'lfs' and/or to llapi.

            ofaaland Olaf Faaland added a comment - Hi Mike, Yes, this is about adding functionality to 'lfs' and/or to llapi.

            Olaf, so this is more about adding such functionality to 'lfs' tool, right? I mean changelog stats

            tappro Mikhail Pershin added a comment - Olaf, so this is more about adding such functionality to ' lfs ' tool, right? I mean changelog stats

            Andreas, yes, this makes sense.

            tappro Mikhail Pershin added a comment - Andreas, yes, this makes sense.

            ofaaland, FYI with newer Lustre it is possible to register users with a name instead of just an ID using "lctl changelog_register -u NAME" so that consumers can determine their own changelog ID. It is also possible to register each user with a specific mask with "-m MASK", so that the changelog itself only contains records that the current users are actually using:

            # lctl --device testfs-MDT0000 changelog_register -u test -m CLOSE
            # lctl get_param mdd.*.changelog_users
            mdd.testfs-MDT0000.changelog_users=
            current_index: 0
            ID                            index (idle) mask
            cl2-test                          0 (4) mask=MARK,CLOSE
            

            Mike, what I did see is that "lctl --device changelog_deregister -u test" works to deregister this user, but only the username doesn't work on the client:

            # lfs changelog_clear testfs-MDT0000 test 1
            lfs changelog_clear: cannot purge records for 'test': Invalid argument (22)
            # lfs changelog_clear testfs-MDT0000 cl2-test 1
            #
            

            This should probably be usable with just the username?

            adilger Andreas Dilger added a comment - ofaaland , FYI with newer Lustre it is possible to register users with a name instead of just an ID using " lctl changelog_register -u NAME " so that consumers can determine their own changelog ID. It is also possible to register each user with a specific mask with " -m MASK ", so that the changelog itself only contains records that the current users are actually using: # lctl --device testfs-MDT0000 changelog_register -u test -m CLOSE # lctl get_param mdd.*.changelog_users mdd.testfs-MDT0000.changelog_users= current_index: 0 ID index (idle) mask cl2-test 0 (4) mask=MARK,CLOSE Mike, what I did see is that " lctl --device changelog_deregister -u test " works to deregister this user, but only the username doesn't work on the client: # lfs changelog_clear testfs-MDT0000 test 1 lfs changelog_clear: cannot purge records for 'test': Invalid argument (22) # lfs changelog_clear testfs-MDT0000 cl2-test 1 # This should probably be usable with just the username?
            ofaaland Olaf Faaland added a comment -

            Hi Mikhail,
            Thanks, I am aware of the changelog_users file. Sounds like I need to update the description a bit - my point is that it's not currrently possible to get that information on a lustre client. Right now one must get to the server to obtain those values. What are your thoughts?

            ofaaland Olaf Faaland added a comment - Hi Mikhail, Thanks, I am aware of the changelog_users file. Sounds like I need to update the description a bit - my point is that it's not currrently possible to get that information on a lustre client . Right now one must get to the server to obtain those values. What are your thoughts?

            People

              tappro Mikhail Pershin
              ofaaland Olaf Faaland
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: