[LU-9023] Second opinion on MDT inode recovery requested

• Type: Question/Request
• Resolution: Unresolved
• Priority: Major
• Assignee: Gabriele Paciucci (Inactive)
• Reporter: Peter Bortas

Description

This is a sanity check question. NSC sees no reason the method described below should not work, but due to the high impact a failure would have we'd like a second opinion. We have scheduled downtime to execute it Thursday next week, 26 Jan.

To sort out the fallout of LU-8953 (out of inodes on ZFS MDT solved by adding more disks to the pool) we need to recreate the original pool. The reason we ran out of inodes was that when the vendor sent us hardware for the latest expansion that was supposed to be equivalent to the last shipment the SSD had switched from reporting 512b blocks to 4k blocks. Since I had not hardcoded ashift we ended up with 6-8 times fewer inodes and this was missed in testing.

There isn't enough slots in the MDSs to solve this by throwing HW at it as a permanent solution, so I need to move all data from pools with ashift=12 to ashift=9. Do you see any problem with just doing the following:

(The funny device names come from running LVM just to get more easily identifiable names)

Unmount the filesystem on all nodes then run something like this for each mdt that needs fixing:

umount lustre-mdt0/fouo6
zfs snapshot lustre-mdt0/fouo6@copythis
zpool create lustre-mdt-tmp -o ashift=9 mirror \
/dev/new_sdr/mdt_fouo6new_sdr \
/dev/new_sdu/mdt_fouo6new_sdu
zfs send -R lustre-mdt0/fouo6@copythis | zfs recv lustre-mdt-tmp/fouo6tmp
zpool destroy lustre-REMOVETHIS-mdt0
zpool create lustre-mdt0/fouo6 -o ashift=9 \
mirror /dev/mds9_sdm/mdt_fouo6_sdm /dev/mds9_sdn/mdt_fouo6_sdn \
mirror /dev/mds9_sdo/mdt_fouo6_sdo /dev/mds9_sdp/mdt_fouo6_sdp
zfs send -R lustre-mdt-tmp/fouo6tmp@copythis | zfs recv lustre-mdt0/fouo6
mount -t lustre lustre-mdt0/fouo6 /mnt/lustre/local/fouo6
zpool destroy lustre-mdt-tmp

The "REMOVETHIS-" inserted due to desktop copy buffer paranoia should be removed before running of course.

Issue Links

• is related to: LUDOC-161 document backup/restore process for ZFS backing filesystems

Comments

[Gabriele Paciucci (Inactive)]

Hi zino,
I would say maybe 3 or 4 days

BTW I'm London based, so we can organize a call to double check the procedure.

[Zhiqi Tao (Inactive)]

Hi Peter B,

Gabriele will try this procedure in an internal develop lab and update this ticket with his experience. We understand your timing, "We have scheduled downtime to execute it Thursday next week, 26 Jan."

We should have results in one day or two.

Best Regards,
Zhiqi

[Peter Bortas]

Appriciated Joseph,

That doc in my mind confirms that we are on the right track with this procedure. I'll wait for Zhiqi to see if he has any further insight.

(And thanks Peter, I can see the tickets again now.)

Cheers,
Peter B

[Peter Jones]

Just a test to check access for zino

[Joseph Gmitter (Inactive)]

Peter,
While we wait for Zhiqi to comment, you can also see the commentary in LUDOC-161 as a 2nd source of information.
Thanks.
Joe

[Joseph Gmitter (Inactive)]

Hi Zhiqi,

Do you have any recommendation here?

Thanks.
Joe

[Peter Bortas]

That create line is incorrect. Should be just "zpool create lustre-mdt0" without the extra filesystem part.