https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7585 Lustre <p>There at occasions where the OI file can become corrupted (e.g. double mount of a filesystem), or otherwise need to be rebuilt (e.g. upgrade of normal ZFS filesystem to Lustre MDT in the future). It would be useful to be able to verify and/or rebuild the ZFS OI files during dnode iteration. </p> LU-7585

Implement OI Scrub for ZFS

Improvement Critical Resolved Fixed nasf Andreas Dilger lfsck Sun, 20 Dec 2015 23:50:49 +0000 Wed, 25 Nov 2020 23:50:23 +0000 Tue, 6 Feb 2018 05:04:00 +0000 Lustre 2.9.0 Lustre 2.11.0 0 7 <p>There have been a number of requests to be able to use tar or rsync or similar tools to do file level backup/restore for ZFS MDTs, for example upgrading the MDT from ldiskfs to ZFS. The only currently supported mechanism for ZFS backup/restore is <tt>zfs send</tt> and <tt>zfs recv</tt> which ties the data to always be in a ZFS filesystem. In addition to implementing ZFS Scrub to repair the OI files in case of corruption (for whatever reason), we should <em>consider</em> whether this would allow us to use tar to restore an ldiskfs MDT into a new ZFS filesystem and/or restore a ZFS tar backup? We would need to support IGIF FIDs for ZFS which I think is relatively low complexity, as well as the ability to rebuild FID-in-dirent which may be considerably more complex.</p> <p>With Data-on-MDT it might also be possible to take an existing regular ZFS (or ext4) filesystem and convert it to a Lustre MDT (<tt>mkdir ROOT/; mv "*" ROOT/; umount; mount -t lustre; run LFSCK</tt>), but this is not a primary goal at this point.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28607" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28607</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> scrub: general framework for OI scrub<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d41e7422086c0ffbe91de18924625eef94448f4f</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28608" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28608</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: FS uuid for zfs backend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bfde4149fc407ea875255ed0c9d691a47b62aadc</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28609" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28609</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: repair FID-in-dirent for ZFS backend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 001cc223bb90a17bb23bf196b264be78d9224e2a</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28610</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: OI scrub for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 77f89863da826af795c6c66618e825fc4e6a5d81</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28678" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28678</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: maintain index file size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b8f1a9a9401bf92f1982a5707878477a803d7141</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28679" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28679</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: move LAST_ID OI mapping out of oi.xx<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8c93eafa7ad984709174048cc5bdf77c540ec5da</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28833</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> utils: new mkfs option 'restore'<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cabff40245dd0580a3676006c3781c1d4cb404f0</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28834" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28834</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: backup index object with plain format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aa57e51ccf31e3aff29e6cae6a5b71fb1b704d2a</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28847" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28847</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> ldiskfs: backup index object with plain format<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 136d925c505bf72ffd2c8da1093b84ed71663640</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28855" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28855</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: create agent object for remote entry<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e22cd933455e777fc2de1170239b56315add1891</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29241" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29241</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: handle non 4K-aligned blocksize<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8ff5673f50d4f4e5bf908d78f6aab87f1e94a728</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29617</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: create agent entry for remote entry<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1186da38066b7139347685d4f3260137d7fdf161</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29696</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> osd: handle PFID EA in LMA properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ae7008f67345203828e39c6db0d4070c01a3a00e</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29710" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29710</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> ldiskfs: auto scrub control<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 66291c9352833186026a24ff69086af513a9ebdd</p> <p>All the patches for this ticket:<br/> 1) <a href="https://review.whamcloud.com/#/c/28607" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28607</a><br/> 2) <a href="https://review.whamcloud.com/#/c/29710" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29710</a><br/> 3) <a href="https://review.whamcloud.com/#/c/29241" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29241</a><br/> 4) <a href="https://review.whamcloud.com/#/c/28679" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28679</a><br/> 5) <a href="https://review.whamcloud.com/#/c/28609" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28609</a><br/> 6) <a href="https://review.whamcloud.com/#/c/29696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29696</a><br/> 7) <a href="https://review.whamcloud.com/#/c/28855" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28855</a><br/> 8) <a href="https://review.whamcloud.com/#/c/29984" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29984</a><br/> 9) <a href="https://review.whamcloud.com/#/c/29617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29617</a><br/> 10) <a href="https://review.whamcloud.com/#/c/29985" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29985</a><br/> 11) <a href="https://review.whamcloud.com/#/c/28610" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28610</a><br/> 12) <a href="https://review.whamcloud.com/#/c/28834" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28834</a><br/> 13) <a href="https://review.whamcloud.com/#/c/28847" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28847</a><br/> 14) <a href="https://review.whamcloud.com/#/c/30106" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30106</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28607/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28607/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> scrub: general framework for OI scrub<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7e21ea59edcc65426e1753ab1884b9a2b93826ce</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29710/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29710/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> osd-ldiskfs: auto scrub control<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3ff1c932f869c8535aaebc56f0c924a760d1d601</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30909" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30909</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: OI scrub for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b5f4bb73612dc2060208c20265f7c4d4ec367371</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30909/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30909/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> zfs: OI scrub for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 89ead218ebe99a955afc0bc7f6aba83ef35019fb</p> <p>Can this work now be marked as resolved?</p> <p>No, there are still three patches to be landed:<br/> <a href="https://review.whamcloud.com/#/c/30910/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30910/</a><br/> <a href="https://review.whamcloud.com/#/c/30911/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30911/</a><br/> <a href="https://review.whamcloud.com/#/c/30106/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30106/</a></p> <p>Isn't that distinct functionality tracked under a different JIRA ticket (<a href="https://jira.whamcloud.com/browse/LU-10193" title="Index object on-disk layout compatibility" class="issue-link" data-issue-key="LU-10193"><del>LU-10193</del></a>)?</p> <p><a href="https://jira.whamcloud.com/browse/LU-10193" title="Index object on-disk layout compatibility" class="issue-link" data-issue-key="LU-10193"><del>LU-10193</del></a> is one of the sub-tasks of this ticket.</p> <p>Ah. Got it. Thanks</p> <p>All the remaining three patches have been approved by reviewers, just waiting gate keeper for landing.</p> <p>so should <a href="https://review.whamcloud.com/#/c/28834/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28834/</a> and <a href="https://review.whamcloud.com/#/c/28847/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28847/</a> be abandoned?</p> <p>Right, abandoned already.</p> <p>ok. So now everything is landed AFAICT</p> <p><del>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch:</del> <a href="https://review.whamcloud.com/40762" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40762</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-7585" title="Implement OI Scrub for ZFS" class="issue-link" data-issue-key="LU-7585"><del>LU-7585</del></a> tests: enable sanity-lfsck tests for ZFS</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: bc6992fe8c019a2525d19496b395bfc5f8815155</del></p> Related LU-7797 LU-3569 LU-5855 LU-7582 LU-8521 LUDOC-161 LU-8620 LU-10186 LU-10187 LU-10188 LU-10189 LU-10190 LU-10192 LU-10193 Development Rank 1|hzxwfb: Rank (Obsolete) 9223372036854775807