https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LUDOC-161 Lustre Documentation <p>The Lustre backup/restore documentation describes how to do a backup/restore using both "tar" and "dd" for ldiskfs devices. However, the preferred mechanism for backing up and restoring ZFS filesystems is via the "<tt>zfs send</tt>" and "<tt>zfs recv</tt>" mechanism. This will preserve the ZFS dnode numbers, and all of the FID-&gt;dnode mappings in the OI files. The zfs dump/restore functionality can also be used to do incremental backups, and keep two physically-separate devices in relatively close synchronization for disaster-recovery and other purposes.</p> <p>It is also not currently possible to do a file-level backup from ldiskfs and restore this into a new ZFS filesystem because there is no OI Scrub facility for ZFS OSDs, so that should be documented as unsupported until <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> is resolved. </p> LUDOC-161

document backup/restore process for ZFS backing filesystems

Improvement Major Resolved Fixed Lustre Manual Triage Andreas Dilger release Thu, 20 Jun 2013 22:24:05 +0000 Fri, 13 May 2022 00:09:38 +0000 Fri, 13 May 2022 00:09:38 +0000 0 13 <p>Here are my notes for a backup and restore test.</p> <h1><a name="ZFSsnapshotsandsend%2Freceiveforobjectbackups."></a>ZFS snapshots and send/receive for object backups.</h1> <p>This example is for combined mgs/mdt object, but the same would apply for an OST device-level backup. This example was run on on Redhat Enterprise Linux 6.2 and lustre 2.4.0. </p> <h2><a name="Serversandfilesystemsintheexample"></a>Servers and filesystems in the example</h2> <p><b>luste2-8-25</b> - MGS/MDT server<br/> lustre-meta/meta - Lustre ZFS MGS/MDT volume/filesystem on lustre2-8-25<br/> <b>lustre2-8-11</b> - OSS/OST server<br/> lustre-ost0 - Lustre ZFS OST volume on lustre2-8-11</p> <h2><a name="Backinguptheobject"></a>Backing up the object </h2> <h3><a name="Takeasnapshot"></a>Take a snapshot</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs snapshot -r lustre-meta@lustre-meta-backup </pre> </div></div> <p>"-r" means do a recursive snapshot, so this will include both the volume and the filesystem.</p> <h3><a name="listexistingsnapshots"></a>list existing snapshots</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lustre2-8-25 ~]# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT lustre-meta@lustre-meta-backup 0 - 30K - lustre-meta/meta@lustre-meta-backup 0 - 287M - </pre> </div></div> <h3><a name="sendandstoreonaremoteZFS%2FLustreserver%3A"></a>send and store on a remote ZFS/Lustre server:</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs send -R lustre-meta@lustre-meta-backup | ssh lustre2-8-11 zfs receive lustre-ost0/lustre-meta </pre> </div></div> <p><b>note</b> "-R" recursively sends the volume, filesystem, and preserves all properties. It is critical to preserve filesystem properties. If not using the "-R" flag, be sure to use "-p", we will show that during recovery.</p> <h3><a name="Examineonremoteside"></a>Examine on remote side</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lustre2-8-25 ~]# ssh lustre2-8-11 zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT lustre-ost0/lustre-meta@lustre-meta-backup 0 - 64.5K - lustre-ost0/lustre-meta/meta@lustre-meta-backup 0 - 605M - </pre> </div></div> <h2><a name="Recoveryfromfailure."></a>Recovery from failure. </h2> <p>In testing, I first corrupted the filesystem with 'dd'. You could also simply reformat it for testing.</p> <h3><a name="CreateanewZFSlustrevolume%2Ffilesystemwiththesamename."></a>Create a new ZFS lustre volume/filesystem with the same name. </h3> <p>In my test case we have a raid 10:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mkfs.lustre --fsname=cove --mgs --mdt --backfstype=zfs lustre-meta/meta mirror ssd0 ssd1 mirror ssd2 ssd3 </pre> </div></div> <h3><a name="Mountwith%22servicestartlustre%22"></a>Mount with "service start lustre"</h3> <p>This makes a volume called "lustre-meta" and filesystem "meta"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lustre2-8-25 ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT lustre-meta 315M 732G 30K /lustre-meta lustre-meta/meta 315M 732G 287M /lustre-meta/meta </pre> </div></div> <p>"mount" command shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lustre-meta/meta on /mnt/lustre/local/cove-MDT0000 type lustre (rw) </pre> </div></div> <h3><a name="Loginintolustre2811%28remotetargetwhereyoustoredthesnapshot%29%2Candsendthefilesystemback."></a>Login into lustre2-8-11 (remote target where you stored the snapshot), and send the filesystem back. </h3> <p>Now I will only send the filesystem back, not the whole volume (why do a whole volume? Convenient if you have multiple datasets?)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs send -p lustre-ost0/lustre-meta/meta@meta-backup | ssh lustre2-8-25 zfs receive lustre-meta/meta-new@recover </pre> </div></div> <p>-p = preserve attributes, important for the lustre filesystem to mount.</p> <h3><a name="BackonOnlustre2811%28failedmetadataserver%29%2Crenamethefilesystemtomakethesnapshotactive."></a>Back on On lustre2-8-11 (failed metadata server), rename the filesystem to make the snapshot active.</h3> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs rename lustre-meta/meta lustre-meta/meta-old cannot rename 'lustre-meta/meta': dataset is busy </pre> </div></div> <p><b>oops!</b> That didn't work. You need to unmount the filesystem so it isn't busy.<br/> Note, this doesn't mean stop the lustre service, if you do you can't access the zfs volume.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>umount /mnt/lustre/local/cove-MDT0000 zfs rename lustre-meta/meta lustre-meta/meta-old zfs rename lustre-meta/meta-new lustre-meta/meta zfs destroy lustre-meta/meta-old service lustre stop service lustre start </pre> </div></div> <p><b>You should now be recovered.</b></p> <p>After living with this for a while, there are obviously a lot of things to consider and possibly add to any documentation on backup/restore for zfs backed filesystems.</p> <p>While snapshots do work, zfs send performance can be very slow, and there is an impact on the file system.</p> <p>More than just a zfs snapshot/send type recipe, some best practice recommendations are needed.</p> <p>Sure Andreas, I will sync up with Nathaniel and Zhiqi.</p> <p>Hi Tom,<br/> Based on your LUG presentation today about using ZFS send/recv for migration, the process to do a backup of the ZFS MDT (or OST). Would you be interested to take a crack st updating the Lustre user manual to document the procedure to do a device level backup of a ZFS MDT or OST? It should be noted that rsync is not a workable solution for ZFS until the ZFS OI scrub is implemented. </p> <p>Hi Andreas,</p> <p>I would be glad to take a crack at updating the Lustre user manual to document the procedure to do a device level backup of a ZFS MDT or OST. </p> <p>The steps for backup ZFS backend var ZPL by 'tar':</p> <p>1) Before umount the target (MDT or OST), please enable index backup on the target. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param osd-zfs.${fsname}-MDT0000.index_backup=1 </pre> </div></div> <p>Or you can write the proc interface on the target directly:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo 1 &gt; /proc/fs/lustre/osd-zfs/${fsname}-MDT0000/index_backup </pre> </div></div> <p>2) Umount the target.</p> <p>3) Import the pool for the target if it is exported during the step 2). For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zpool import lustre-mdt1 [-d ${target_device_dir}] </pre> </div></div> <p>4) Enable "canmount" property on the target FS. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs set canmount=on ${fsname}-mdt1/mdt1 </pre> </div></div> <p>You also can specify the "mountpoint" property. By default, it will be /${fsname}-mdt1/mdt1</p> <p>5) Mount the target as 'zfs'. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs mount ${fsname}-mdt1/mdt1 </pre> </div></div> <p>6) Backup the data. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar jcf /tmp/target.tar.bz2 --xattrs --xattrs-include="trusted.*" -C /${fsname}-mdt1/mdt1/ . </pre> </div></div> <p>7) Umount the target and export the pool if you want.</p> <p>Please save the /tmp/target.bz2 and /tmp/target.ea. If you want to migrate system from ZFS to ldiskfs, please backup the system as the same steps.</p> <p>The steps for restore ZFS backend via ZPL by 'tar':</p> <p>1) Create new pool for the target if necessary, then reformat new Lustre FS with "--replace" parameter. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mkfs.lustre --mgs --fsname=${fsname} --mdt --index=0 --replace --param=sys.timeout=20 --backfstype=zfs --reformat ${fsname}-mdt1/mdt1 ${target_device} </pre> </div></div> <p>2) Enable "canmount" property on the target FS. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs set canmount=on ${fsname}-mdt1/mdt1 </pre> </div></div> <p>3) Mount the target as 'zfs'. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs mount ${fsname}-mdt1/mdt1 </pre> </div></div> <p>4) Retore the data. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar jxf /tmp/target.tar.bz2 --xattrs --xattrs-include="trusted.*" -C /${fsname}-mdt1/mdt1/ </pre> </div></div> <p>5) Remove stale OIs and index objects. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cd /${fsname}-mdt1/mdt1 &amp;&amp; rm -rf oi.* OI_* lfsck_* LFSCK &amp;&amp; sync &amp;&amp; cd - </pre> </div></div> <p>6) Umount the target.</p> <p>7) (optional) If the restored system has different NID as the backup system, please change NID. For detail, please refer to Lustre manual 14.5. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mount -t lustre -o nosvc ${fsname}-mdt1/mdt1 $mount_point lctl replace_nids ${fsname}-MDTxxxx $new_nids ... umount $mount_point </pre> </div></div> <p>8) Mount the target as "lustre". Usually, we will use "-o abort_recov" option to skip unnecessary recovery. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mount -t lustre -o abort_recov ${fsname}-mdt1/mdt1 $mount_point </pre> </div></div> <p>The osd-zfs can detect the restore automatically when mount the target, then trigger OI scrub to rebuild the OIs and index objects asynchronously at background. You can check the OI scrub status. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl get_param -n osd-zfs.${fsname}-${target}.oi_scrub </pre> </div></div> <p>Or you can read the proc interface on the target directly:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cat /proc/fs/lustre/osd-zfs/${fsname}-${target}.oi_scrub </pre> </div></div> <p>If you want to restore the system from ldiskfs-based backup, please follow the same steps.</p> <p>If you want to migrate the system from ldiskfs to ZFS, please backup your system as following:</p> <p>1) Before umount the target (MDT or OST), please enable index backup on the target. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param osd-ldiskfs.${fsname}-MDT0000.index_backup=1 </pre> </div></div> <p>Or you can write the proc interface on the target directly:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo 1 &gt; /proc/fs/lustre/osd-ldiskfs/${fsname}-MDT0000/index_backup </pre> </div></div> <p>2) Umount the target.</p> <p>Then you can follow the standard steps to backup the target normally via "tar" and "getfattr" as described in the Lustre manual 18.3.</p> <p>Please consider to update Lustre manual according to above three comments. Thanks!</p> <p><a href="https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213782&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213782" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213782&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213782</a></p> <p><a href="https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213784&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213784" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213784&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213784</a></p> <p><a href="https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213786&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213786" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LUDOC-161?focusedCommentId=213786&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-213786</a></p> <p>Joseph Gmitter (joseph.gmitter@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30172" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30172</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> zfs: Backup/Restore of ZFS backend filesystems<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 843d957d7553ade81404dd3fdbefc6240b32d11a</p> <p>Joseph Gmitter (joseph.gmitter@intel.com) merged in patch <a href="https://review.whamcloud.com/30172/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30172/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LUDOC-161" title="document backup/restore process for ZFS backing filesystems" class="issue-link" data-issue-key="LUDOC-161"><del>LUDOC-161</del></a> zfs: Backup/Restore of ZFS backend filesystems<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3eba9c153757b350882c28161e45b4e5815617b2</p> <p>Included in the 2.11 manual.</p> Related LU-7585 LUDOC-56 LU-8124 LU-9023 Development Rank 1|hzvtq7: Rank (Obsolete) 8773