[LU-17107] Seeking Advice on Efficient Data Migration Methods

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.15.2
• Type: Question/Request
• Priority: Minor
• Reporter: BNL Team
• Assignee: Colin Faber
• Resolution: Unresolved

Environment

RHEL8

• Severity: 3

Description

Hi,

In our last data migration efforts involving 50 OSS servers, we observed slow migration speeds even though we used a lot of concurrent threads with the lfs_migrate command. To address this, we switched to the cp and mv methods later, which greatly enhanced our migration speed.

The migration command we initially used is as below:

lfs find --ost ost_name /mount/point | lfs_migrate -y

 

As we prepare for our next migration task, I'd like to know if there are better migration methods we haven't considered yet.

Thank you for your guidance,

Jane

Comments

[Colin Faber]

Hi,

https://hpc.github.io/mpifileutils/ might be a useful tool for this task and help speed up your migration dramatically.

[BNL Team]

Hi,

Thanks for the link. I checked mpifileutils but only saw that it can be used to migrate from one lustre to another lustre. What I need is a fast approach to migrate files from a bunch of retiring OSTs to other active OSTs. Is there a way to do it using mpifileutils? If yes, how to do it? 

Thanks,

Jane

[Colin Faber]

If you have a specific layout defined, dsync from mpifileutils can be leveraged to copy into these targets.