[LU-17316] make more changelog metadata visible on clients

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.15.3
• Type: Improvement
• Priority: Minor
• Reporter: Olaf Faaland
• Assignee: Mikhail Pershin
• Resolution: Unresolved
• Labels: llnl
• Severity: 3

Description

For an application running on a lustre client consuming changelogs, I believe the following would be useful. Both of these are available on the MDS via lctl get_param but there is currently no way (that I'm aware of) to get this information on a client node.

On Lustre servers, we avoid running software other than critical management and monitoring tools, for both security and reliability reasons. I believe that is common practice. That means that something like a policy engine should be able to get whatever changelog information it needs on a lustre client (not server) node. Our local concern at the moment is Starfish, but I think below would be equally useful for others.

1. The ability to obtain the list of registered changelog users for a given MDT.

Currently, the application needs the user to specify which changelog user (e.g. "cl2") to use. For cases where the user is only running one application which consumes changelogs, and that application interacts with only one user, this isn't necessary (I suspect this is most cases, but I may be wrong).

In addition, the application currently can't confirm a changelog user is valid. One example we've seen is that it would be useful to be able to detect that the changelog user was deregistered while the application was using it.

2. The ability to learn the next available changelog record ID and the last changelog record ID for a given MDT.

The application could adapt to the number of outstanding changelog records by creating more consumer threads when then are many changelogs available, and reducing thread count when there are few. The application could also better handle errors, such as when changelogs are cleared by a sysadmin to free space while the application is running.

It's possible to obtain the next changelog record if there is one which hasn't yet been cleared, with "lfs changelog | head -n1" or similar. But if there is no changelog record yet to be cleared, that technique doesn't work.

Issue Links

• is related to: LU-15373 changelog improvements tracking
• is related to: LU-17290 Don't deregister idle changelog consu...

Comments

[Olaf Faaland]

I'm posting this here to find out if others agree these capabilities would be useful, and to get a sense of how hard this might be for Eric or Gian to work on.

[Colin Faber]

Hi tappro can you review and comment?

[Mikhail Pershin]

both items can be done already by issuing 'lctl get_param mdd.*.changelog_users' on server. Its output gives all you've asked for:

mdd.lustre-MDT0000.changelog_users=
current_index: 1872821234
ID                            index (idle) mask
cl1                      1871194397 (117)

it would list all registered users with their names (IDs) and here you can find current_index which is common value and means the latest used index, while all users have own index as well and that is the oldest unprocessed index, so difference between current_index and the least user index would give amount of unprocessed records in changelog.

But I suspect you meant not core Lustre ability to report such info but support for that from some tool (is it one you've mentioned as Starfish?)

[Olaf Faaland]

Hi Mikhail,
Thanks, I am aware of the changelog_users file. Sounds like I need to update the description a bit - my point is that it's not currrently possible to get that information on a lustre client. Right now one must get to the server to obtain those values. What are your thoughts?

[Andreas Dilger]

ofaaland, FYI with newer Lustre it is possible to register users with a name instead of just an ID using "lctl changelog_register -u NAME" so that consumers can determine their own changelog ID. It is also possible to register each user with a specific mask with "-m MASK", so that the changelog itself only contains records that the current users are actually using:

# lctl --device testfs-MDT0000 changelog_register -u test -m CLOSE
# lctl get_param mdd.*.changelog_users
mdd.testfs-MDT0000.changelog_users=
current_index: 0
ID                            index (idle) mask
cl2-test                          0 (4) mask=MARK,CLOSE

Mike, what I did see is that "lctl --device changelog_deregister -u test" works to deregister this user, but only the username doesn't work on the client:

# lfs changelog_clear testfs-MDT0000 test 1
lfs changelog_clear: cannot purge records for 'test': Invalid argument (22)
# lfs changelog_clear testfs-MDT0000 cl2-test 1
#

This should probably be usable with just the username?

[Mikhail Pershin]

Andreas, yes, this makes sense.

[Mikhail Pershin]

Olaf, so this is more about adding such functionality to 'lfs' tool, right? I mean changelog stats

[Olaf Faaland]

Hi Mike,
Yes, this is about adding functionality to 'lfs' and/or to llapi.

[Olaf Faaland]

Andreas,
Thanks, I'd forgotten that users could name changelog users. I agree, that helps. It would still be useful to have a mechanism to confirm that a changelog user (user-named or not) is valid, and allowing 'lfs' to report the list would be one way of doing that.

[Andreas Dilger]

Olaf, there was also LU-17290 filed that requested a "more graceful" handling of idle changelog users when they need to be evicted because of too many unprocessed records.

Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"?

I'm not against your proposal to export more changelog information to the client, but also want to ensure that these requests are implemented in a way that is aligned.

[Olaf Faaland]

Andreas,

> Would that help your cause if your changelog consumer always used the same name, and didn't get fully deregistered if it stops processing records but still returned an error if it was "evicted"?

Yes, that would help.

[Olaf Faaland]

To clarify, we don't run things like Starfish or Robinhood on the lustre server nodes themselves, for security, performance, and reliability reasons (possibly we're making the wrong choices, but that's the line of reasoning). So exporting more changelog information to the client potentially allows such services to run better.

Also, I'm not asking Whamcloud do this work. I'm just trying to make sure the proposal is sound, and to get a sense of how experienced a developer would need to be to have any hope of getting it done. Low granularity is fine for my purposes.