[LU-15419] Separate Quota Master from MDT0000

Related — Tue, 25 Apr 2023 20:24:56 +0000

It would be useful to separate the Quota Master service from (only) running on MDT0000, in order to allow the filesystem to continue working if MDT0000 is temporarily unavailable (or eventually has been removed from the filesystem completely).

The main functions that the Quota Master service provides are:

storing per-user/group/project/pool quota ID limits. These limits are relatively static and can be replicated to one or more additional MDTs in case of a failure, to avoid their loss. This is independently useful because these limits are specified by the administrator and cannot be automatically recovered if lost.
allocating chunks of a quota ID's limit (qunit) to individual MDT/OST targets for them to track against their local filesystem's quota usage. These qunit assignments can very dynamic when files are created/written, but could be recovered directly from the quota accounting tables on the targets themselves if the primary copy is lost.

Beyond increased availability, distributing subsets of the quota IDs across all MDTs would also have additional benefits:

distribution of load over multiple MDS/MDT for handling qunit assignments, since this is independent for different quota IDs
less "oversubscription" of load/recovery if one MDT is unavailable, since only a fraction of the quota workload would need to be taken over by another MDS/MDT if it becomes unavailable.

Initially, replicating the quota limits to one or more MDTs would provide a useful incremental benefit. Similarly, recovering quota usage from the targets is useful by itself, regardless of where the Quota Master is running.

For a distributing quota ID handling across all MDTs, a mapping of IDs to MDTs is needed. A simple mechanism would be like "mdt_idx = (id % mdt_count)", but suffers from a number of issues:

ID-to-MDT mapping changes significantly if mdt_count changes
does not tolerate gaps in mdt_index (i.e. if MDT0004 is missing)
One option would be to use CRUSH or a similar algorithm to map quota IDs to the available MDTs.

Whamcloud Community JIRA

[LU-15419] Separate Quota Master from MDT0000