Motivation

Following the presentation of `parallel_tools.py` and its benchmark results at the conference LAD26, this proposal describes a standalone C utility for large-scale migration between Lustre OSTs.

The goal is to extend the existing `lustre-migrate-file` C helper with parallel scheduling and job management, allowing administrators to evacuate selected OSTs and control migration load without requiring the Python scheduler.

Reference to the C- helper: https://github.com/jguzdv/parallel-tools

Proposed functionality

• Parallel migration of multiple files using a configurable worker pool.
• Targeted evacuation of selected OSTs, with control over new mirror placement.
• Mirror management, including establishing a usable mirror outside the excluded OSTs before removing affected mirrors.
• Verification of the final layout and requested mirror count.
• Support for paths beyond `PATH_MAX` through descriptor-based path handling.
• Runtime adjustment of worker count and batch size.
• Retries for transient failures.
• Machine-readable per-file results identifying completed, failed, and unfinished work.

Layout support and data protection

The utility should support migration of both mirrored files and files protected by erasure coding, including evacuation of selected OSTs while preserving the configured protection scheme.

Migration must use layout-specific procedures:

For mirrored layouts, establish a usable mirror outside the excluded OSTs before removing affected mirrors.

For erasure-coded layouts, relocate or reconstruct affected data and parity fragments while maintaining recoverability throughout the operation.

Verify the resulting placement and the required redundancy before reporting success.

Existing data or parity fragments must not be removed until the replacement state has been validated. Unsupported layouts or operations must be reported explicitly without silently reducing data protection.

Implementation will require identifying the Lustre versions and APIs that support erasure-coded layout inspection, migration, reconstruction, and validation. Checkpoint and recovery must account for partially completed operations for both protection schemes.

Checkpoint and resume

On a checkpoint request, the program would stop dispatching new work and let active operations reach safe stopping points. It would then save the configuration, pending work, and migration progress to a JSON file and exit.

A later invocation would load the checkpoint, validate file identities and migration state against the current filesystem, and continue unfinished work. The checkpoint must account for input paths that have not yet been processed, so resumption does not depend on the original input producer still running.

Implementation

The existing C helper provides the per-file migration logic using the native Lustre API. Development would focus on:

• Moving the migration-related scheduler functionality from Python into C.
• Replacing process-wide mutable state with per-job contexts.
• Removing working-directory changes.
• Reviewing the thread-safety of the Lustre API calls and their dependencies.
• Implementing checkpoint storage and recovery.

Validation

Testing would cover concurrent migrations, file changes during migration, transient failures, and checkpoint recovery. Performance comparisons would include parallel `lfs migrate` runs under equivalent conditions, building on the benchmarks presented at the conference.

Integration question

Would the preferred upstream direction be a separate utility distributed with the Lustre client packages, or integration into the existing `lfs` tools?