Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
Lustre 2.10.5
-
None
-
Multi-tenancy, several filesets per node
-
9223372036854775807
Description
We're finding some sort of limitation in our particular use case for Lustre where we export Lustre to many different VLANs with many filesets. Because the VLANs are directly connected to the servers (the servers' interfaces are connected to trunk ports on the switches) we need to specify, for each VLAN, a different set of NIDs for each device. If we have, for instance, 10 VLANs, that means 10 NIDs for each server.
Besides, if we export several filesets to the same client, this means we need several LNETs on the same client (e.g. tcp10 for fileset1, tcp11 for fileset2, tcp12 for fileset3). This would mean 3 NIDs per VLAN.
That means that in the hypothetic configuration above we would need to define 30 NIDs for each server and at least 60 NIDs for each device (failover NIDs). Currently Lustre supports around 20 NIDs per device, above this number the configuration will silently fail and only after trying to unsuccessfully mount the Lustre client we could realize that the config in the device is truncated after around 20 NIDs.
Lustre routers can help to scale with problem 1 (multiple VLANs) but they cannot help if we have multiple filesets per client (maximum of 20 filesets per client in a routed environment).
It would be a good improvement if this limit could be increased now that new Lustre use cases are in play.