https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3488 Lustre <p>I am testing recovering a failed metadata server via zfs snapshot. </p> <p>Everything seems to go as expected, but when trying to mount the filesystem, either via the /etc/init.d/lustre start script I get an error about the backing filesystem:</p> <p><span class="error">&#91;root@lustre2-8-25 ~&#93;</span># mount -vt lustre lustre-meta/meta /mnt/lustre/local/cove-MDT0000<br/> arg<span class="error">&#91;0&#93;</span> = /sbin/mount.lustre<br/> arg<span class="error">&#91;1&#93;</span> = -v<br/> arg<span class="error">&#91;2&#93;</span> = -o<br/> arg<span class="error">&#91;3&#93;</span> = rw<br/> arg<span class="error">&#91;4&#93;</span> = lustre-meta/meta<br/> arg<span class="error">&#91;5&#93;</span> = /mnt/lustre/local/cove-MDT0000<br/> source = lustre-meta/meta (lustre-meta/meta), target = /mnt/lustre/local/cove-MDT0000<br/> options = rw<br/> checking for existing Lustre data: not found<br/> mount.lustre: lustre-meta/meta has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool</p> <p>I can manually mount it directly (posix I guess) to the filesytem with 'zfs mount' - and things look about right. </p> <p>Have tried pawing through the init scripts but haven't found anything.</p> <p>This could perhaps be a documentation note, and I'm missing something easy, so that's why I'm filing this as a 'story'. I see there are some issues open for zfs lustre documentation. Just don't know what to do, information is scarce.</p> <p>Or, if this isn't supported there's always 'dd'? </p> <p>Scott</p> RHEL6.2 LU-3488

zfs mdt backup/recovery via snapshot

Story Minor Closed Not a Bug WC Triage Scott Nolin Thu, 20 Jun 2013 20:42:58 +0000 Mon, 24 Jun 2013 15:16:38 +0000 Mon, 24 Jun 2013 15:16:38 +0000 Lustre 2.4.0 0 5 <p>I don't think "dd" backups would work for ZFS. The recommended backup method for ZFS (which I now realize is probably not documented in the Lustre manual, and I filed <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> for this) is to use "<tt>zfs dump</tt>" and "<tt>zfs restore</tt>".</p> <p>That said, I think it should be possible to mount a ZFS snapshot, assuming that the primary version of the filesystem is not mounted.</p> <p>It might be useful to run <tt>mount.lustre</tt> directly under gdb or other debugger or via strace to see where the the error is coming from. It appears that this is failing in <tt>osd_is_lustre-&gt;zfs_is_lustre</tt>, so either it cannot open the dataset, or the dataset parameters were not copied with the snapshot? It may be that this relates to <a href="https://jira.whamcloud.com/browse/LU-2190" title="failure on conf-sanity.sh test_49: Different LDLM_TIMEOUT:6 20 20" class="issue-link" data-issue-key="LU-2190"><del>LU-2190</del></a> <a href="http://review.whamcloud.com/5220" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5220</a>.</p> <p>Is "zfs dump" some extra lustre specific utility? I've never encountered that one elsewhere with standard zfs, and the zfs command and man page on my lustre 2.4 install don't indicate it's an option.</p> <p>Scott</p> <p>Andreas, perhaps you meant <tt>zfs send</tt> and <tt>zfs receive</tt>?</p> <p>Scott, can you elaborate a bit more on what you mean by "I am testing recovering a failed metadata server via zfs snapshot"? I've run into the issue where the "osd is zfs" check fails before, but I can't recall offhand what the problem was or how I fixed it. I think I opened a ticket on that, give me a minute to search for it.</p> <p>EDIT: This appears to be the ticket I was thinking of <a href="https://jira.whamcloud.com/browse/LU-2333" title="Cannot mount a MGS device backed by ZFS if &quot;--fsname&quot; was not passed to mkfs.lustre" class="issue-link" data-issue-key="LU-2333"><del>LU-2333</del></a>. Make sure the properties are correct on the dataset you are trying to mount.. What does "<tt>sudo zfs get all lustre-meta/meta | grep lustre:</tt>" say?</p> <p>Prakash,</p> <p>I took a snapshot of my zfs backed mgs/mdt (combined device), and used send/receive to save on another server. I then corrupted my mdt filesystem, rebuilt the filesystem, and attempted to use send/receive with the snapshot to recover it. "service lustre start" fails the mount, I then tried the mount command by hand that I listed above. I'll list my notes from that below.</p> <p>Anway, looking at the properties with "zfs get all" as you suggested does show that the "lustre:" properties are missing. I can try setting those with "zfs set" - the only one I'm not sure of is lustre:flags. My OST's all have "34".</p> <p>Here are my notes from my test process:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- Make snapshot zfs snapshot -r lustre-meta@meta-backup - send and store entire volume and filesystem (-R) on another zfs server (oss in this case) zfs send -R lustre-meta@meta-backup | ssh lustre2-8-11 zfs receive lustre-ost0/lustre-meta - Rebuilt the MDS filesytem. I went to the oss holding the snapshot, and put it in a gzip file, so I could just work with it locally on the mds. (on oss) zfs send lustre-ost0/lustre-meta/meta@meta-backup | gzip &gt; /root/meta-snap.gz So to recover (on ost) gunzip -c /root/meta-snap.gz | zfs receive lustre-meta/meta-new@recover I noticed it builds a mount in /lustre-meta/meta-new that we don't want umount /lustre-meta/meta-new rm -rf /lustre-meta/meta-new zfs rename lustre-meta/meta lustre-meta/meta-old zfs rename lustre-meta/meta-new lustre-meta/meta zfs destroy lustre-meta/meta-old service lustre stop service lustre start </pre> </div></div> <p>I tried setting the lustre: properties with 'zfs set' and it didn't seem to help. As a WAG I tried flags of "34" and "100" (based on what I saw elsewhere).</p> <p>So the zfs get results now show:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lustre-meta/meta lustre:fsname cove local lustre-meta/meta lustre:mgsnode 172.16.24.12@o2ib local lustre-meta/meta lustre:flags 34 local lustre-meta/meta lustre:version 1 local lustre-meta/meta lustre:index 0 local lustre-meta/meta lustre:svname cove-MDT0000 local </pre> </div></div> <p>It's not important that I recover this filesystem, I just want to work out a procedure. So will gladly test any ideas or confirm what should work (perhaps to help with the documentation effort).</p> <p>Scott</p> <p>Scott,</p> <p>You can preserve the properties by adding the -p option to 'zfs send'. Alternately you can use the -R option for zfs send which will create a replication stream and preserve everything. See the 'zfs send' section of the zfs(8) man page.</p> <p>Try the "-p" option when sending the backup to your gzip archive. I tried this locally in a VM, and it worked for me.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs send -p lustre-ost0/lustre-meta/meta@meta-backup | gzip &gt; /root/meta-snap.gz </pre> </div></div> <p>This will preserve the original data set properties, so you don't have to set them manually later on. Side note, you also need to do this in the original send of the dataset to backup, but "-R" automatically does this for you.</p> <p>Thank you Brian and Prakash.</p> <p>I used "-R" in my initial zfs send, but failed to use that or the -p flag in my send to gzip. </p> <p>I simply resent (ssh to the meta this time) with the proper '-p' flag and now all appears well, lustre is recovering.</p> <p>I couldn't get it to mount by setting properties - the "lustre:flags" on the now mounted filesystem is "37". I wonder if that was the issue, I have no insight into the flags property.</p> <p>So this is resolved.</p> <p>For what it's worth, I'll write up my procedure and add it to a comment on <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> in case that helps.</p> <p>Thanks again,<br/> Scott</p> <p>My bad, yes I meant "zfs send" and "zfs receive". Glad you figured out the details with sending the pool parameters. </p> Development Epic/Theme zfs Rank 1|hzvtpj: Rank (Obsolete) 8770