Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20344

Reachability-scoped peer discovery

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Note: the following should be considered a draft design. I want to extend this to both add/del cases in the discovery merge, and I think we should deprecate lnet_peer_discovery_disabled and use the lnet_peer_discovery_mode to cover disabled (equivalent to lnet_peer_discovery_disabled=1), enabled (lnet_peer_discovery_disabled=0) as well as the new mode.

      Background

      When LNet discovers a peer, the merge step adds every NID from the
      peer's ping reply into the local peer tables. This includes NIDs on
      networks this node has no interface on and no route to. On large,
      heterogeneous, or multi-tenant fabrics this means a node accumulates
      peer NIDs it can never use. The unreachable NIDs:

      • pollute the local peer tables,
      • are considered by multi-rail path selection, wasting cycles on paths
        that can never succeed, and
      • generate health/recovery churn as LNet repeatedly probes NIDs that
        are unreachable from this node.

      The motivating deployment is cloud / multi-tenancy. There, a server on
      the storage fabric should retain full visibility of every NID in the
      network, while a client should only learn the NIDs it has locally
      configured or can reach via a configured route. It has no business
      building peer state for NIDs that belong to other tenants' networks
      and that it cannot reach.

      Enhancement

      Add a global, opt-in LNet tunable, lnet_discovery_mode, that scopes
      the discovery merge to reachable NIDs:

      value mode a discovered NID on network N is merged when...
      0 full (default) always – current behavior, unchanged
      1 reachable N is a local network or this node has a configured route to N

      Because the policy keys on reachability, the desired server/client
      asymmetry falls out automatically: a server sitting on the fabric
      reaches every NID and keeps full visibility, while a client on a
      narrower network (e.g. tcp-only) only adopts the NIDs it can actually
      reach. A tcp client that routes to o2ib via tcp<->o2ib routers will
      still build multi-rail peers from the peer's o2ib NIDs (a route
      exists) while ignoring every other off-net NID.

      The default (full) is byte-for-byte the current behavior, so the
      change is opt-in and carries no regression risk for existing setups.

      Semantics

      • Receive-side policy only. The mode changes what a node merges from a
        ping reply; it does not change what a node advertises or negotiate
        anything on the wire. Only the node applying the policy needs the
        feature – peers can run stock LNet.
      • Block-new-only. Enabling reachable stops adding unreachable
        NIDs; it does not retroactively prune NIDs learned earlier under
        full.
      • Advertised primary handling. If the peer's advertised primary NID is
        itself unreachable, the node keeps its existing reachable primary and
        still merges the reachable NIDs, rather than adopting an off-net
        primary.
      • Manually configured peers are exempt. DLC-configured peer NIDs do
        not carry the discovery flag and are never filtered.

      Configuration

      The tunable is wired through the usual interfaces:

      • lnetctl set discovery_mode <0|1>
      • lnetctl global show reports discovery_mode
      • DLC YAML import (lnetctl import)
      • sysfs: /sys/module/lnet/parameters/lnet_discovery_mode

      Scope

      Plan to land to master and backport to supported branches.

      Test

      New sanity-lnet.sh subtest (requires >= 1 remote node):
      the remote configures NIDs on two networks; the local node configures
      only the first. Under discovery_mode=1 the unreachable NID is
      filtered from the peer after lnetctl discover; under discovery_mode=0
      it is merged, confirming the default path is unchanged. Additional test (requires >= 2 remote nodes) exercises the feature in a routed configuration.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: