https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11512 Lustre <p>DL-SNAP is a feature designed for directory level file backups.</p> <p>It is implemented on top of lustre ldiskfs without modification of ext4 disk format, though we set a special original flag to FEATURE of the superblock in order to check whether DL-SNAP is enabled.</p> <p>The feature makes execution of e2fsck command failed although no modification to ext2 file system structure.</p> <p> </p> <p>DL-SNAP uses COW(Copy On Write) mechanism to reduce backup time and storage usage.  When we create a snapshot, only inodes are created. At this point, new data blocks are not allocated on OST.  Data blocks of snapshot are allocated when original file is modified.</p> <p> </p> <p>Not only root users but also ordinary users can create snapshots. Users can create a snapshot of a directory using lfs command with snapshot option, and restore files by usual process such as cp command.</p> <p> </p> <p>For more information, please check the <a href="http://wiki.lustre.org/images/8/80/LUG2016D2_DL-SNAP_Sumimoto.pdf" class="external-link" target="_blank" rel="nofollow noopener">presentation at LUG2016</a>.</p> LU-11512

DL-SNAP: Directory Level Snapshot

New Feature Minor Open Unresolved Tatsushi Takamura Tatsushi Takamura patch Fri, 12 Oct 2018 10:05:57 +0000 Mon, 12 Jul 2021 18:29:41 +0000 Lustre 2.6.0 0 13 <p>This ticket has the "patch" label, but I don't see a patch attached to this ticket or in Gerrit?</p> <p>Sorry for the late reply. I have attached dl_snapshot patch to this ticket.</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31390/31390_dl_snapshot.patch" title="dl_snapshot.patch attached to LU-11512">dl_snapshot.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Tatsushi Takamura (takamr.tatsushi@jp.fujitsu.com) uploaded a new patch: <a href="https://review.whamcloud.com/33555" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33555</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11512" title="DL-SNAP: Directory Level Snapshot " class="issue-link" data-issue-key="LU-11512">LU-11512</a> ldiskfs: DL-SNAP Directory Level Snapshot<br/> Project: fs/lustre-release<br/> Branch: b2_6<br/> Current Patch Set: 1<br/> Commit: 303f237c5e116ac271f03db3cc1a3aeb2568b7f0</p> <p>Still not look into the detailed codes, but from PPT, a quick question:</p> <p>Is DL-SNAP recursive for sub dir?</p> <p>Thank you for contributing the patch. Is there any kind of design documentation or description that was written before or during development? Even if it is in Japanese, we can try machine translation to try and understand the implementation better.</p> <p>I've been looking into the Ext4 patch a little bit. Please correct me if I am wrong Tatsushi-san, but I think on Ext4 level, recurisve subdirectory snapshot is not supported. It might be implemented on Lustre level? I have a feeling that data on MDT might causes some difficulty for this snapshot implementation.</p> <p>We didn't understand that adding original code to ldiskfs is not acceptable, because there are some patches(lustre/kernel_patches/patches/) modifying original kernel source.<br/> This DL-SNAP code is impossible to be merged into Ext4 mainline and it is impossible to realize DL-SNAP feature without adding code to ldiskfs.<br/> So we think it is impossible to be merged into lustre.</p> <p>For future reference, could you tell us the rules(which sources are acceptable or not to modify), difference in contents or polices of "ldiskfs/kernel_patches/patches/" and "lustre/kernel_patches/patches/".<br/> For example, can we create patches(not merged in mainline) which modify kernel code in "lustre/kernel_patches/patches/"?</p> <p>Tatsushi-san, it is not impossible to get changes included into the upstream ext4, but it takes time, effort, and coordination to do so. It is unfortunate that this patch is already several years old, as we could have used that time to discuss such changes with the upstream ext4 maintainers (of which I am one) and work to include the changes into the upstream ext4 code over that time. Maybe the changes could have already been in the upstream kernel (even RHEL or SLES distro kernels), as was done with project quotas, large xattrs, and large directory support. </p> <p>The patches under <tt>lustre/kernel_patches/patches</tt> are for the Lustre server kernel. While we <b>do</b> allow patches to be included there, they cannot be required patches, only optional patches (eg. to improve performance or to backport fixes and features from upstream kernels). We have moved away from adding features only to the Lustre kernel, since this adds maintenance burden for each new kernel added, and makes it difficult for users to update their kernels. The current Lustre server can build and run without any <b>server</b> patches, though an ldiskfs patch series is still needed for each kernel, but not if ZFS is used.</p> <p>I will take a look at the DL Snap changes when I have a chance, and see whether they are acceptable to be submitted upstream for ext4. At a minimum, they would need to be updated to the latest kernel, but there may be other changes needed in order to make them compatible with newer features. </p> <p>Thank you very much for teaching me in great detail, Andreas.<br/> We think DL_SNAP patch is difficut to be merged into Ext4 mainline, because it don't work as a stand-alone.<br/> But, could you please look at this patch if that is okay. If you need design documentations still, we will send you.</p> <p>Hello Wang and Li</p> <p>DL-SNAP can create snapshot files recursivly for subdirectory. <br/> It scans files under the target directory and sends snapshot creation requests correspond to the original files(for the number of files times).</p> <p>HAS_SNAPSHOT feature has been being reserved too long without the feature implementation.  <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> guarded the flag in EXT4 today, but at some moment without any progress, the flag can be excluded.</p> Related Development Rank 1|i00453: Rank (Obsolete) 9223372036854775807