[LU-18885] manage large NID tables on MGS with multiple networks

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.17.0
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Mikhail Pershin

Description

Multi Tenancies setups can provide very large NID lists from target with many networks configured. Currently they are stored as plain list per target in MGS memory which is hard to parse them later to select particular network only. So right now it is easy to store such lits, as they are just copied but hard to filter out while reading. Especially considering that nidtable is updated once but then each client reads it to get updated info, so that is "single update -> concurrent read * number of clients"

The idea is to organize NID lists in MGS at per-network basis, with separate list per each configured network. All incoming NIDs from target are parsed and stored in such manner in memory, then IR fetches them by network it needs without parsing, all at once. By doing that network filtering is done during nidtable update - once target does that, but then multiple concurrent reads are fetching new data without need to filter NIDs out each time

I am thinking about using Xarray for that storing NID lists with network ID as index

Issue Links

• is related to: LU-10360 use Imperative Recovery logs for client->MDT/OST connections