[LU-14634] mechanism to exclude tasks from generating Changelog records

Related — Wed, 15 Dec 2021 08:46:44 +0000

Some filesystem modifications from processes such as HSM Copytools, data movement agents, etc. shouldn't generate Changelog records, since they are themselves consumers of the Changelog. If their filesystem operations generate new Changelog records, then this generates more records that they need to process (hopefully not creating further work for themselves).

It should be possible in some manner to "opt out" of generating Changelog records on specific nodes for specific processes, in a manner that is controlled by the MDS itself. It shouldn't be possible for processes to arbitrarily opt out of generating Changelog records by themselves, since that would allow bypassing system auditing. Similarly, whole-node exclusion by itself potentially risks missing important operations that are done by non-agent processes on a server-mounted client (e.g. NFS or CIFS server running on a server-mounted client, admin copying important files from the server into the filesystem and then not generating archive/mirroring events for those files).

This can be achieved for whole-node exclusion with nodemap.*.audit_mode=0. A more fine-grained approach might be to allow an audit_exclude_gid=GID parameter to allow only processes with a specific GID to be excluded from generating Changelog records. Alternately, allowing an audit_exclude_jobid=JobID on nodes in the nodemap would avoid the need to depend on a specific GID (which may be needed for file access permissions), but potentially at the expense of security since a process JobID could be set arbitrarily.

Alternately, the agent processes could use a specific JobID value for their filesystem-modifying operations, and then exclude that JobID from further processing when seen in the logs. This allows a "self contained" exclusion to be implemented to reduce the processing of those records, but doesn't reduce the overhead of creating and storing the Changelog records in thee first place.

Whamcloud Community JIRA

[LU-14634] mechanism to exclude tasks from generating Changelog records