[LU-1651] Audit multireader support changelogs

• Status: Closed
• Project: Lustre
• Type: Bug
• Priority: Major
• Reporter: Robert Read (Inactive)
• Assignee: John Hammond
• Resolution: Fixed

Sub-Tasks

Key	Summary	Type	Status	Assignee
LU-1652	Please inspect the HLD for changelog ...	Technical task	Resolved	Robert Read

• Key: Summary
• Summary: Type
• Type: Status
• Status: Assignee
• Severity: 3

Description

We need to confirm that changelogs supports multiple readers like it is supposed. Changelog records need to be retained until all readers have seen them, and stale readers need to be evicted at some point so we can flush the logs.

Comments

[nasf (Inactive)]

Changelog supports a set of interfaces for recording lustre namespace changes, reading such records, purging specified record(s), and so on. Currently, all such records will not be purged automatically, even if someone has read such record(s) already (maybe for replicate the lustre system). Means such records can be read many times before they are purged explicitly.

If you think some old records are not useful (since you have replicated current lustre system already) or you just want to free up some space, then you can purge some records as following:
"lfs changelog_clear <mdtname> <id> <endrec>"
Such command indicates that changelog records up to <endrec> are no longer of interest to consumer <id>, removing them. An <endrec> of "0" means removing all records.

On the other hand, when all changelog users deregistered, then changelog will be off and all records will be purged automatically under such case, even though no one used them yet. (I am not sure it is reasonable, but it is the current implementation)

[nasf (Inactive)]

Lustre implements a small rsync tools – lustre_rsync, which uses changelog as the backend support. When lustre_rsync runs, it will purge (by calling ioctl with "OBD_IOC_CHANGELOG_CLEAR" explicitly) the old changelog of 100 records each time periodically.

[Robert Read (Inactive)]

So the feature to automatically flush log records once they've been consumed by all readers has not been implemented, and lustre_rsync is a destructive reader of changelogs. This is very helpful, thanks.

[nasf (Inactive)]

Changelog only supplies the interfaces for recording/reading lustre namespace changes. It is the HSM engine's duty to decide (according to the so called 'policy') which change(s) should be recorded and how long the record(s) should be retained, just like lustre_rsync does.

[Robert Read (Inactive)]

Right, the application decides when to mark the records as "consumed," but if there is more than one reader then records should not be deleted until all of the readers have marked them as "consumed."

[nasf (Inactive)]

Unfortunately, the current implementation of changelog has no idea about "consumed", it neither knows nor cares about how many reader want to access the record(s). But the application can keep the "consumed" conception by keeping the next record index to be read. Anyway, even if some reader had not accessed related changelog record(s) yet, he can not prevent others to purge such record(s) explicitly.

[Robert Read (Inactive)]

Can you estimate what needs to be done to add state to changelogs to allow keeping track of multiple registered readers and to only flush records that have been marked consumed by all the readers? This will require maintaining persistent state for each client, and we'll need options for allow stale readers to be "evicted" if the logs reach a certain size or after a period of time of inactivity from that client. We'll also need the ability to stop updates on the filesystem of the log fills up to support, say, a high availability replication application.

[nasf (Inactive)]

This is basic idea for changelog enhancement. We can implement it within one month. There are some uncertain issues mentioned in the document, need more discussion.

[John Hammond]

Each registered changelog reader has a persistent minimum record index (mcud_minrec) which can be updated by calling lfs changelog_clear. This also purges changelog records with index lower than the all of the minimum record indexes.

So this is fixed and the ticket can be closed.