Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-1187

DNE Phase 1: Remote Directories

    Details

    • Type: Epic
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Lustre 2.4.0
    • Fix Version/s: Lustre 2.4.0
    • Labels:
      None
    • Epic Name:
      DNE Phase 1
    • Rank (Obsolete):
      3974

      Description

      DNE Phase 1: Remote Direcories

      This issue is for tracking progress of DNE Phase 1: Remote Directories.

      Introduction

      Today's HPC systems routinely configure Lustre with tens of thousands of clients. Each client draws resources from a single metadata server (MDS). The efforts of the Lustre community continue to improve the performance of the MDS, however, as client numbers continue to increase the single MDS represents a fundamental limit to filesystem scalability.

      The goal of the Distributed Namespace (DNE) project is to a deliver a documented and tested implementation of Lustre that addresses this scaling limit by distributing the filesystem metadata over multiple metadata servers. This is an ambitious engineering project that will take place over a period of two years. It requires considerable engineering and testing resource and will therefore be performed in the two phases described below.

      Phase 1: Remote Directories

      This phase introduces a useful minimum of distributed metadata functionality. The purpose primarily to ensure that efforts concentrate on clean code restructuring for DNE. The phase focuses on extensive testing to shake out bugs and oversights not only in the implementation but also in administrative procedures. DNE brings new usage patterns that must necessarily adapt to manage multiple metadata servers.

      The Lustre namespace will be distributed by allowing directory entries to reference sub-directories on different metadata targets (MDTs).  Individual directories will remain bound to a single MDT, therefore metadata throughput on single directories will stay limited by single MDS performance, but metadata throughput aggregated over distributed directories will scale.

      The creation of non-local subdirectories will initially be restricted to administrators with a Lustre-specific mkdir command.  This will ensure that administrators retain control over namespace distribution to guarantee performance isolation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                laisiyao Lai Siyao
                Reporter:
                rhenwood Richard Henwood (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: