https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7607 Lustre <p>During migration, the MDT FID of the migrated file is changed to reflect the new MDT the inode is stored on. However, it would be possible to keep the user-visible inode constant after migration by storing the original FID into the LMA as a new field. If present, this saved FID could be used to generate the inode number for userspace instead of the current FID so that it doesn't affect user tools such as backups. </p> LU-7607

Preserve inode number after MDT migration

Improvement Minor Open Unresolved Lai Siyao Andreas Dilger dne3 Thu, 24 Dec 2015 10:37:57 +0000 Wed, 4 Oct 2023 18:28:26 +0000 Lustre 2.8.0 0 12 <p>Di, what do you think about this idea? It should be possible to add this as a "compat" feature to the LMA only if the inode is migrated. Since the internal references are all using the FID, the inode number displayed by "ls" and "stat" don't really matter.</p> <p>Hmm, as long as we do not use this original FID for internal references, it should be ok. As for backup, are you trying to temporarily resolve <a href="https://jira.whamcloud.com/browse/LU-6866" title="MDT file migration is incompatible with HSM" class="issue-link" data-issue-key="LU-6866"><del>LU-6866</del></a>? i.e. copy tool will always use this original FID as the identifier to locate the file in the archive? then we probably need record this original FID into changelog, or the copy tool needs to be changed to retrieve FID from LMA?</p> <p>I hadn't thought about keeping the whole FID, just the original inode number for exposure to userspace for tools like tar or other backup tools that depend on the inode number. Since we won't be using this number internally, only for exposure via stat() and readdir() it doesn't matter if it is different than the real FID.</p> <p>Do you think there is value to keep the original FID?</p> <p>For the HSM FID problem I think a different solution is needed. Instead of storing the FID in the archive, it is better to store the archive identifier (UUID or whatever) in the Lustre inode as a part of the composite layout. That allows storing multiple versions of the file in the archive, as well as allowing partial HSM file restore with composite files. </p> <p>Actually I am a bit confused now. Since migration will keep namespace consistency, so either stat() or readdir() should return the real (correct) ino. I do not know why should we keep the original ino (or FID) after migration? I probably miss sth. Could you please explain the purpose of keeping consistent ino here? why these external tool needs consistency ino? Thanks.</p> <p>The point of keeping a consistent inode number is that some tools, such as backups, depend on the inode number to remain the same so they can do incremental backups. Otherwise, they can't tell the difference between migrate changing the inode number, or the file being deleted and a new file created with the same name. NFS servers in userspace would also use the inode number. </p> <p>NFS file handles generated in the kernel by Lustre are the same, but since we encode the FID into the Lustre file handle this wouldn't help - we'd need to allow the original FID to be looked up on the original MDT with a redirection to the new FID. </p> <p>I see. Thanks. Hmm for stat() we only need fill the original FID into mdt_body of getattr request, and cache it in ll_inode_info, and fill it to stat-&gt;ino, But for readdir(), it will read from the directory entries directly, if we inject LMA original ino checking in this process, it might slow down the readdir a lot. </p> <p>Lai, I recall not too long ago we discussed the ability to save the old FID after migration. Is there anything that needs to be updated in this ticket to describe your proposal?</p> <p>IMO the overhead of reserving inode number is quite high, and rather than saving the original FID, I'd prefer to add an option to keep file inode untouched in migration, that is to say, for existing sub files, inode won't be migrated, but namespace updated.</p> <p>If we still prefer migrating inode, I'd suggest drop the support of preserving inode number, but add FID mapping to support NFS export: a special OI file will be added, which contains mappings from original FID to new FID, and in lu_object_find() it will lookup this<br/> OI file, if new FID is found, it's replied client with -EREMOTE, and client will resend the request with new FID to the correct MDT, for some request like rename, client may need to retry several times if more than involved file are migrated. There will be a garbage collect thread on server to remove aged mapping from this OI file.</p> <p>Could we emit the changes out the changelog?  If someone wants to keep track of changes of fid/inode through time, they could have a listener set up to catch/archive them.</p> <p>The various calls to find a FID, etc. could simply call up to the userspace service to get historical info.</p> <p>Ben, there is a already a <tt>MIGRT</tt> ChangeLog record for inode migration. </p> <p>Lai, it definitely makes sense to have an option to migrate the parent directory and filenames without migrating the file inodes. In that case there is no need for this feature to preserve the inode numbers, since they won't change. </p> <p>This is only needed in the case where the inode is moved to a new MDT, which can be needed in case of removing an MDT, or if an MDT is very full, not in the normal space balancing case. </p> <p>I think storing the original FID in the inode is not too hard, and it will always be unique. Adding the old and new FID in the OI table is also useful. We can't return -EREMOTE to NFS clients, but it can be handled by the Lustre client so that it doesn't return -ESTALE to NFS. </p> <p>Andreas, okay.</p> <p>I was looking at whether we could use the FID stored in the LOV EA to preserve the "original" inode number of the file. It seems that the FID is stored in the LOV EA in each component and is also preserved over OST and MDT migration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tests# lfs setstripe -E 1M -L mdt -E 1G -c 3 -E eof /mnt/testfs/dir1/tt tests# dd if=/dev/zero of=/mnt/testfs/dir1/tt bs=1M count=2 tests# lfs path2fid /mnt/testfs/dir1/tt [0x200001b72:0x10c07:0x0] tests# lfs getstripe -v /mnt/testfs/dir1/tt components: - lcme_id: 1 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_seq: 0x200001b72 lmm_object_id: 0x10c07 lmm_fid: [0x200001b72:0x10c07:0x0] lmm_stripe_count: 1 - lcme_id: 2 lcme_extent.e_start: 1048576 lcme_extent.e_end: 1073741824 sub_layout: lmm_seq: 0x200001b72 lmm_object_id: 0x10c07 lmm_fid: [0x200001b72:0x10c07:0x0] - lcme_id: 3 lcme_extent.e_start: 1073741824 lcme_extent.e_end: EOF sub_layout: lmm_seq: 0x200001b72 lmm_object_id: 0x10c07 lmm_fid: [0x200001b72:0x10c07:0x0] tests# lfs migrate -c 3 /mnt/testfs/dir1/tt tests# lfs getstripe -v /mnt/testfs/dir1/tt lmm_seq: 0x200001b72 lmm_object_id: 0x10c07 lmm_fid: [0x200001b72:0x10c07:0x0] lmm_stripe_count: 3 tests# lfs migrate -m1 /mnt/testfs/dir1 tests# lfs path2fid /mnt/testfs/dir1/tt [0x240001b70:0x2743:0x0] tests# lfs getstripe -v /mnt/testfs/dir1/tt lmm_seq: 0x200001b72 lmm_object_id: 0x10c07 lmm_fid: [0x200001b72:0x10c07:0x0] lmm_stripe_count: 3 </pre> </div></div> <p>There is a bug (<a href="https://jira.whamcloud.com/browse/LU-13426" title="&quot;lfs migrate&quot; on DoM component clobbers LOV EA FID" class="issue-link" data-issue-key="LU-13426"><del>LU-13426</del></a>) if there is a DOM component in the layout that clobbers the FID, but DOM migration is relatively new and can be fixed to preserve the FID properly.</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch <a href="https://review.whamcloud.com/38135" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38135</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7607" title="Preserve inode number after MDT migration " class="issue-link" data-issue-key="LU-7607">LU-7607</a> dne: add FID map interfaces<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 9<br/> Commit: 2119206d30f6d84ac1974d9c7d3b24bc25eee1e4</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch <a href="https://review.whamcloud.com/38233" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38233</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7607" title="Preserve inode number after MDT migration " class="issue-link" data-issue-key="LU-7607">LU-7607</a> dne: support FID map<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 7<br/> Commit: 0f7af1befc6de458acd84b817c535938f839c808</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch <a href="https://review.whamcloud.com/38285" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38285</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7607" title="Preserve inode number after MDT migration " class="issue-link" data-issue-key="LU-7607">LU-7607</a> mdd: add fidmap reclaim thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 4<br/> Commit: 3068abb9c2e5f85eb76f9e94c7469a761af5f86a</p> <p>Add in links to existing patches under this ticket, not sure why they weren't previously created. </p> Related LU-6866 LU-2430 LU-13426 LU-14975 LU-7749 LU-14465 LU-11753 LU-11306 LU-11025 Development Epic Link EX-4481 Rank 1|hzxwrb: Rank (Obsolete) 9223372036854775807