[LU-11510] preserve PFL/FLR/DoM layout with lfs_migrate

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.11.0, Lustre 2.12.0
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.5
• Type: Bug
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: Emoly Liu
• Resolution: Fixed
• Severity: 3

Description

The lfs_migrate tool does not currently preserve the PFL/FLR/DoM components of the layout when a file is migrated. By default, it will get the stripe_count and stripe_size of the file (maybe pool and mirror_count if https://review.whamcloud.com/32977 is updated to do so), but it does not preserve more complex layout details.

Dropping the initial components of a PFL file is probably desirable, since it is unlikely that a file will become smaller, and it makes sense to keep the stripe_count of a file at the widest needed.

We do need to keep the number of mirrors across migration, and it would probably be good to keep any OST pools per mirror, if they exist.

Separately, it would be useful to have some way to migrate only the mirror(s) that exist on particular OSTs if we are trying to empty particular OSTs.

Issue Links

• is related to: LU-10801 argument parsing in lfs_migrate is br...
• is related to: LU-9982 Clients striping from mapped FID in n...
• is related to: LU-9392 lfs migrate -o and lfs setstripe -o s...
• is related to: LU-13703 fix lfs_migrate to work with PFL argu...
• is related to: LU-9629 lfs migrate does not work as a non-ro...

Comments

[Emoly Liu]

adilger,

To meet the requirement, the following changes will be done to lfs_migrate tool:

• use yaml file to save migrated file layout
• save the detailed OST pool information

What's more, do we need to add options to control these features or by default? 

[Andreas Dilger]

I think the easiest way to implement this is to leverage the "lfs setstripe --copy" option to copy the layout from the source file to the target file. It may be that with PFL/FLR/DoM files that we should change lfs_migrate to not try to get the source stripe_count and stripe_size and just copy the layout from the source file (ie. do less handling of the layout rather than more)?

[Gerrit Updater]

Emoly Liu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36082
Subject: LU-11510 scripts: add --copy and --yaml to lfs_migarate
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 16fb4f3eb169c05809784d14f54cf3fd4f67fd36

[Andreas Dilger]

It seems like the core problem is that "lfs migrate" doesn't handle PFL files very well. Starting with a 2-component stripe_count=<1, 4> file (with the 4-stripe component initialized):

# lfs getstripe !$
lfs getstripe /myth/tmp/pfl/64M
/myth/tmp/pfl/64M
  lcm_layout_gen:  3
  lcm_entry_count: 2
    lcme_id:             1
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   33554432
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       1
      lmm_layout_gen:    0
      lmm_stripe_offset: 2
      lmm_pool:          audio
      lmm_objects:
      - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x26cdc7:0x0] }

    lcme_id:             2
    lcme_flags:          init
    lcme_extent.e_start: 33554432
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  4
      lmm_stripe_size:   1048576
      lmm_pattern:       1
      lmm_layout_gen:    0
      lmm_stripe_offset: 4
      lmm_objects:
      - 0: { l_ost_idx: 4, l_fid: [0x100040000:0x311e78:0x0] }
      - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x22ca87:0x0] }
      - 2: { l_ost_idx: 1, l_fid: [0x100010000:0x1b1931:0x0] }
      - 3: { l_ost_idx: 3, l_fid: [0x100030000:0x22233d:0x0] }

Using "lfs getstripe -c /myth/tmp/pfl/64M" returns a stripe_count of 4 from the last instantiated component as one would expect. Running "lfs migrate" without any arguments doesn't preserve the PFL layout AND it doesn't even keep the stripe count from the last initialized component of the source file. At a minimum, "lfs migrate /myth/tmp/pfl/64M" should use stripe_count=4 to create the new file, or copy the whole PFL layout but instead it is using the filesystem-wide default stripe_count=1:

# lfs getstripe !$
lfs getstripe /myth/tmp/pfl/64M
/myth/tmp/pfl/64M
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       1
lmm_layout_gen:    4
lmm_stripe_offset: 3
	obdidx		 objid		 objid		 group
	     3	       2237247	     0x22233f	             0

I think it makes sense to use the "lfs migrate --copy" option internally to copy the layout from the source file, or from the parent directory for "lfs_migrate -R". The "--yaml" option still can specify an arbitrary layout. That can potentially simplify migration

[David Stewart]

Hi Emoly/Andreas, do we have any updates on this?

[Gerrit Updater]

Emoly Liu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37401
Subject: LU-11510 lfs: create a comp file correctly when migrating
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a12d86989152bc6fbb9ad3e6566c24c29b077f76

[Andreas Dilger]

the next step is to move the file to the new OSTs by adding an option to lfs_migrate tool to let the customer specify the OST list.

Normally when doing large-scale migration, the OST selection would be handled by the MDS by disabling object creation on some OSTs (via "lctl set_param osp.<fsname>-OST[xxxx-yyyy].max_create_count=0") and running "lfs find -ost xxxx,...,yyyy" to list all of the files on those OSTs to migrate. I think the common case is finding files on OSTs that shouldn't be used, and the MDS QOS policy will already bias file allocation onto OSTs with more free space.

I think would be useful for making easier file layout policies for users are:

• using "lfs setstripe -copy XXX" to select a template layout from an existing file/directory (this should already work with the new patch), but allowing an "alias" like -copy=@mypattern that looks for the file "$HOME/.lfs/mypattern" or similar
• finishing patch https://review.whamcloud.com/28972 "LU-9982 lustre: Clients striping from mapped FID in nodemap" so that it is possible to set a per-fileset default layout
• implementing the LU-9392 functionality to allow setstripe to specify a range of OSTs, but set the stripe count smaller than the OST list, similar to how an OST pool works, but available to users instead of admins

[Emoly Liu]

implementing the LU-9392 functionality to allow setstripe to specify a range of OSTs, but set the stripe count smaller than the OST list, similar to how an OST pool works, but available to users instead of admins

This is close to my idea.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36082/
Subject: LU-11510 lfs: migrate a composite layout file correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8bedfa377fbd0c9f1b6ea2c40d36fdcaa52137df

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37814
Subject: LU-11510 lfs: migrate a composite layout file correctly
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: fdb231d505dda018515f754475527041e67af7c8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37814/
Subject: LU-11510 lfs: migrate a composite layout file correctly
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: dc88cb0843e57be3930558e4e1b5c9b5d3af3311