[LU-3488] zfs mdt backup/recovery via snapshot

• Type: Story
• Resolution: Not a Bug
• Priority: Minor
• Affects Version/s: Lustre 2.4.0
• Environment: RHEL6.2

Epic/Theme

zfs

• Assignee: WC Triage
• Reporter: Scott Nolin (Inactive)

Description

I am testing recovering a failed metadata server via zfs snapshot.

Everything seems to go as expected, but when trying to mount the filesystem, either via the /etc/init.d/lustre start script I get an error about the backing filesystem:

[root@lustre2-8-25 ~]# mount -vt lustre lustre-meta/meta /mnt/lustre/local/cove-MDT0000
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = lustre-meta/meta
arg[5] = /mnt/lustre/local/cove-MDT0000
source = lustre-meta/meta (lustre-meta/meta), target = /mnt/lustre/local/cove-MDT0000
options = rw
checking for existing Lustre data: not found
mount.lustre: lustre-meta/meta has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool

I can manually mount it directly (posix I guess) to the filesytem with 'zfs mount' - and things look about right.

Have tried pawing through the init scripts but haven't found anything.

This could perhaps be a documentation note, and I'm missing something easy, so that's why I'm filing this as a 'story'. I see there are some issues open for zfs lustre documentation. Just don't know what to do, information is scarce.

Or, if this isn't supported there's always 'dd'?

Scott

Comments

[Andreas Dilger]

My bad, yes I meant "zfs send" and "zfs receive". Glad you figured out the details with sending the pool parameters.

[Scott Nolin (Inactive)]

Thank you Brian and Prakash.

I used "-R" in my initial zfs send, but failed to use that or the -p flag in my send to gzip.

I simply resent (ssh to the meta this time) with the proper '-p' flag and now all appears well, lustre is recovering.

I couldn't get it to mount by setting properties - the "lustre:flags" on the now mounted filesystem is "37". I wonder if that was the issue, I have no insight into the flags property.

So this is resolved.

For what it's worth, I'll write up my procedure and add it to a comment on LUDOC-161 in case that helps.

Thanks again,
Scott

[Prakash Surya (Inactive)]

Try the "-p" option when sending the backup to your gzip archive. I tried this locally in a VM, and it worked for me.

zfs send -p lustre-ost0/lustre-meta/meta@meta-backup | gzip > /root/meta-snap.gz

This will preserve the original data set properties, so you don't have to set them manually later on. Side note, you also need to do this in the original send of the dataset to backup, but "-R" automatically does this for you.

[Brian Behlendorf]

Scott,

You can preserve the properties by adding the -p option to 'zfs send'. Alternately you can use the -R option for zfs send which will create a replication stream and preserve everything. See the 'zfs send' section of the zfs(8) man page.

[Scott Nolin (Inactive)]

I tried setting the lustre: properties with 'zfs set' and it didn't seem to help. As a WAG I tried flags of "34" and "100" (based on what I saw elsewhere).

So the zfs get results now show:

lustre-meta/meta  lustre:fsname         cove                   local
lustre-meta/meta  lustre:mgsnode        172.16.24.12@o2ib      local
lustre-meta/meta  lustre:flags          34                     local
lustre-meta/meta  lustre:version        1                      local
lustre-meta/meta  lustre:index          0                      local
lustre-meta/meta  lustre:svname         cove-MDT0000           local

It's not important that I recover this filesystem, I just want to work out a procedure. So will gladly test any ideas or confirm what should work (perhaps to help with the documentation effort).

Scott

[Scott Nolin (Inactive)]

Prakash,

I took a snapshot of my zfs backed mgs/mdt (combined device), and used send/receive to save on another server. I then corrupted my mdt filesystem, rebuilt the filesystem, and attempted to use send/receive with the snapshot to recover it. "service lustre start" fails the mount, I then tried the mount command by hand that I listed above. I'll list my notes from that below.

Anway, looking at the properties with "zfs get all" as you suggested does show that the "lustre:" properties are missing. I can try setting those with "zfs set" - the only one I'm not sure of is lustre:flags. My OST's all have "34".

Here are my notes from my test process:

- Make snapshot

zfs snapshot -r lustre-meta@meta-backup

- send and store entire volume and filesystem (-R) on another zfs server (oss in this case)

zfs send -R lustre-meta@meta-backup | ssh lustre2-8-11 zfs receive lustre-ost0/lustre-meta

- Rebuilt the MDS filesytem. I went to the oss holding the snapshot, and put it in a gzip file, so I could just work with it locally on the mds.

(on oss)
zfs send lustre-ost0/lustre-meta/meta@meta-backup | gzip > /root/meta-snap.gz
 
So to recover (on ost)
 
gunzip -c /root/meta-snap.gz | zfs receive lustre-meta/meta-new@recover

I noticed it builds a mount in /lustre-meta/meta-new that we don't want

umount /lustre-meta/meta-new
rm -rf /lustre-meta/meta-new

zfs rename lustre-meta/meta lustre-meta/meta-old
zfs rename lustre-meta/meta-new lustre-meta/meta
zfs destroy lustre-meta/meta-old

service lustre stop
service lustre start

[Prakash Surya (Inactive)]

Andreas, perhaps you meant zfs send and zfs receive?

Scott, can you elaborate a bit more on what you mean by "I am testing recovering a failed metadata server via zfs snapshot"? I've run into the issue where the "osd is zfs" check fails before, but I can't recall offhand what the problem was or how I fixed it. I think I opened a ticket on that, give me a minute to search for it.

EDIT: This appears to be the ticket I was thinking of LU-2333. Make sure the properties are correct on the dataset you are trying to mount.. What does "sudo zfs get all lustre-meta/meta | grep lustre:" say?

[Scott Nolin (Inactive)]

Is "zfs dump" some extra lustre specific utility? I've never encountered that one elsewhere with standard zfs, and the zfs command and man page on my lustre 2.4 install don't indicate it's an option.

Scott

[Andreas Dilger]

I don't think "dd" backups would work for ZFS. The recommended backup method for ZFS (which I now realize is probably not documented in the Lustre manual, and I filed LUDOC-161 for this) is to use "zfs dump" and "zfs restore".

That said, I think it should be possible to mount a ZFS snapshot, assuming that the primary version of the filesystem is not mounted.

It might be useful to run mount.lustre directly under gdb or other debugger or via strace to see where the the error is coming from. It appears that this is failing in osd_is_lustre->zfs_is_lustre, so either it cannot open the dataset, or the dataset parameters were not copied with the snapshot? It may be that this relates to LU-2190 http://review.whamcloud.com/5220.