https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4293 Lustre <p>"lfs_migrate -y" aborts with this error</p> <p>cannot swap layouts between &lt;filename&gt; and a volatile file (Operation not permitted)</p> <p>This seems to happen for all files. The lfs_migrate operation aborts on the first file.</p> Lustre 2.4.1 RHEL6 2.6.32-358.18.1.el6_lustre.x86_64 LU-4293

lfs_migrate is failing with a volatile file Operation not permitted error

Bug Major Resolved Fixed Bruno Faccini Wolfgang Baudler Fri, 22 Nov 2013 13:42:13 +0000 Fri, 21 Mar 2014 03:40:43 +0000 Fri, 21 Feb 2014 18:52:14 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 0 11 <p>I investigated this on my home system, since I was seeing this problem intermittently as well.</p> <p>It looks like the problem is caused in my case because the problematic files were created with Lustre 1.8 or earlier and have IGIF FIDs (these show as inode number &lt; 4B, ). Migrating files created under 2.x is OK (these have very large inode numbers):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ ls -i /mnt/tmp/tmp/1M 144115440203858997 1024 /mnt/tmp/tmp/1M $ lfs path2fid /mnt/tmp/tmp/1M [0x200003ab4:0x435:0x0] $ lfs migrate -i 1 /mnt/tmp/tmp/1M $ ls -i /mnt/tmp/tmp/4M 2422700 4096 /mnt/tmp/tmp/4M $ lfs path2fid /mnt/tmp/tmp/4M [0x24f7ac:0x2d77b0e5:0x0] $ lfs migrate -i 1 /mnt/tmp/tmp/4M cannot swap layouts between /mnt/tmp/tmp/4M and a volatile file (Operation not permitted) error: migrate: migrate stripe file '/mnt/tmp/tmp/4M' failed </pre> </div></div> <p>This is because there is a check in <tt>mdd_swap_layouts_allowed()</tt> that prevents layout swap for IGIF FIDs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> mdd_layout_swap_allowed(<span class="code-keyword">const</span> struct lu_env *env, struct mdd_object *o1, struct mdd_object *o2) { <span class="code-keyword">const</span> struct lu_fid *fid1, *fid2; fid1 = mdo2fid(o1); fid2 = mdo2fid(o2); <span class="code-keyword">if</span> (!fid_is_norm(fid1) || !fid_is_norm(fid2) || (mdd_object_type(o1) != mdd_object_type(o2))) RETURN(-EPERM); </pre> </div></div> <p>This was done to prevent clients from being able to swap the contents of regular files with internal system files by using their (internal) IGIF FIDs.</p> <p>I'm not yet sure how to detect the difference between filenames that are visible in the namespace and ones that are accessed by IGIF FID. It does seem that "lfs fid2path" and $MOUNT/.lustre/fid/ <em>can</em> detect the difference between IGIF and FID access with my 2.4.1 server. I took a normal FID and then figured out its IGIF FID by looking at the MDT inode's inum/generation directly, and got a "no such file or directory", so that is a good start.</p> <p>However, it also appears that some files in the MDT root directory (e.g. backups of fld, seq_srv, seq_cli, etc) are readable via $MOUNT/.lustre/fid/ and have an IGIF FID assigned to them. The shell also thinks that these FIDs have write permission (i.e. test -w "$MOUNT/.lustre/fid/<span class="error">&#91;0x2686:0xc40fa169:0x0&#93;</span>" returns 0), even though I get a permission denied error trying to modify them, so normal write permission checks will fail. That might be a problem with LFSCK adding these files into the OI when they shouldn't be. The originals of these files correctly have SEQ 0x200000001 and get an error from obf_lookup(), but I think it makes sense to mark all files in the top-level MDT/OST root directory inaccessible, and only add files under ROOT to the OI.</p> <p>In the short term, "lfs_migrate" <em>should</em> fall back to using rsync internally if "lfs migrate" returns an error, but I haven't tested this. It would also be useful to fix the error message printed by "lfs migrate", since I find the current one confusing. I don't think it needs to mention anything about volatile files.</p> <p>I also see that the failure in "<tt>lfs migrate</tt>" is causing volatile inodes to be leaked on the MDT. I see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsck 1.42.7.wc1 (12-Apr-2013) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Unattached inode 116 Connect to /lost+found? no Unattached inode 117 Connect to /lost+found? no Unattached inode 118 Connect to /lost+found? no Unattached inode 409 Connect to /lost+found? no </pre> </div></div> <p>The unattached inodes look like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs 1.42.7.wc2 (12-Apr-2013) debugfs: stat &lt;116&gt; Inode: 116 Type: regular Mode: 0644 Flags: 0x0 Generation: 2124528570 Version: 0x00000001:00001b6d User: 0 Group: 0 Size: 0 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x5296625c:00000000 -- Wed Nov 27 14:21:32 2013 atime: 0x52966270:00000000 -- Wed Nov 27 14:21:52 2013 mtime: 0x52966270:00000000 -- Wed Nov 27 14:21:52 2013 crtime: 0x5296625c:087ac7fc -- Wed Nov 27 14:21:32 2013 Size of extra inode fields: 28 Extended attributes stored in inode body: lma = "00 00 00 00 00 00 00 00 00 04 00 00 02 00 00 00 45 07 00 00 00 00 00 00 " (24) lma: fid=[0x200000400:0x745:0x0] compat=0 incompat=0 link = "df f1 ea 11 01 00 00 00 3a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 22 00 00 00 02 00 00 00 07 00 00 00 01 00 00 00 00 2e 0c 13 14 12 3a 56 4f 4c 41 54 49 4c 45 3a 3a " (58) lov = "d0 0b d1 0b 01 00 00 00 45 07 00 00 00 00 00 00 00 04 00 00 02 00 00 00 00 00 10 00 01 00 01 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 " (56) BLOCKS: </pre> </div></div> <p>where the "<tt>link</tt>" xattr is the "<tt>^.L^S^T^R:VOLATILE</tt>" string.</p> <p>Bruno, does the layout swap failure here have any relation to <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a>? I saw many leaked objects after failures calling "lfs migrate" on an IGIF file.</p> <p>Andreas, <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a> is about how to handle/recover-from any layout swap failures during HSM Restore ops and avoid leaving file in inconsistent state, this ticket is for a specific condition and consequence. I will try to push a patch soon based on your already+very detailled analysis, thanks !!</p> <p>But finally may be it is related to <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a>, after my recent discovery there that the layouts swap-back upon failure may not work as expected!</p> <p>Andreas,<br/> After the failure, is the original file still accessible ?</p> <p>Also, since it appears the main problem for this ticket is that trying "lfs migrate" on 1.8 files with IGIF fids will fail and leak volatile objects on MDT, why not 1st change test in mdd_layout_swap_allowed() by adding/allowing also fid_is_igif() files ? Then we will address the special cases of system/ROOT files and the failure/recovery.</p> <p>Or do you still think all cases must be handled in "lfs migrate" by switching to rsync if not a system/ROOT file, and the leak to be addressed separately ?</p> <p>Uploaded a patch to correctly handle "lfs migrate" failure in lfs_migrate and do fallback to rsync copy+rename:<br/> <a href="http://review.whamcloud.com/8616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8616</a></p> <p>This is not a complete solution to the problem here, but is useful in any case.</p> <p>Bruno, yes after the migration failure the original file is still accessible. Ideally, if the MDD could verify if the object is in the namespace then that would be a good way to know if it can be migrated or not. I think allowing to migrate IGIF FIDs is a potential problem, but not worse than other issues a client might have, so that would be a solution for 2.6 if no others are available.</p> <p>Sorry to be late, but I am back on this one.</p> <p>Andreas, sorry to ask but can you explain me how the files created in MDT root-directory have an IGIF assigned ??</p> <p>I also confirm that as part of <a href="https://jira.whamcloud.com/browse/LU-3834" title="hsm_cdt_request_completed() may clear HS_RELEASED on failed restore" class="issue-link" data-issue-key="LU-3834"><del>LU-3834</del></a>, and fault-injection during layouts-swap to verify patch behavior, I reproduce the volatile object leak (inode links number is 1 and e2fsck detects "Unattached inode") on MDT. In my case, and for one layouts-swap forced error, I see one orphan inode with ".^L^S^T^R:VOLATILE"/LUSTRE_VOLATILE_HDR linkEA but also one with "i_am_nobody", did you also find this ? </p> <p>But anyway, this clearly indicate that there is something to address and fix upon layouts-swap error.</p> <p>I created the IGIF files under 1.8 and upgraded to 2.x. You could also get the same effect by mounting a 2.4 MDT as ldiskfs, deleting the "lma" xattr, then rounding and running LFSCK to fix the OI. </p> <p>Humm thanks, I understand I better had to read <a href="https://jira.whamcloud.com/browse/LU-4392" title="LFSCK shouldn&#39;t assign FIDs, linkEA to files in &quot;/&quot; directory" class="issue-link" data-issue-key="LU-4392"><del>LU-4392</del></a> sub-task and learn more about LFSCK behavior than to ask, sorry !!</p> <p>So now, do we really need to detect such files with a wrongly assigned IGIF by LFSCK (to be fixed in <a href="https://jira.whamcloud.com/browse/LU-4392" title="LFSCK shouldn&#39;t assign FIDs, linkEA to files in &quot;/&quot; directory" class="issue-link" data-issue-key="LU-4392"><del>LU-4392</del></a> sub-task) ?? Because if not, fix for this ticket's original issue could simply be to add fid_is_igif() test for both files having their layouts swapped, in mdd_layout_swap_allowed().<br/> I may miss some special cases about files with IGIF here since you wrote about "internal system files by using their (internal) IGIF FIDs" ??…</p> <p>I will also work on the MDT volatile object leak upon layouts swap failure, may be as part as a new ticket.</p> <p>Patch to allow layout swap for IGIF file is at <a href="http://review.whamcloud.com/8737" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8737</a>.</p> <p>I wonder if the -EPERM error seen on non-IGIF files is due to some file ownership problem like <a href="https://jira.whamcloud.com/browse/LU-3826" title="copytool restore should set owner of volatile file before copy" class="issue-link" data-issue-key="LU-3826"><del>LU-3826</del></a> or similar?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ((attr1-&gt;la_uid != attr2-&gt;la_uid) || (attr1-&gt;la_gid != attr2-&gt;la_gid)) RETURN(-EPERM); </pre> </div></div> <p>This will fail if lfs_migrate is not creating the file with the same ownership as the original file? If lfs_migrate is running as root, then it should be able to fchown() the file after it is created. The actual permissions don't matter, because the temporary file will be deleted, but this is proof that the caller of the migrate has permission to do this swap.</p> <p>Patch <a href="http://review.whamcloud.com/8616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8616</a> was cherry-picked to Lustre b2_5 branch.</p> <p>My patch at <a href="http://review.whamcloud.com/8737" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8737</a> seems to address the original need (allow legitimate IGIFs, handle root special-case, ...) for this ticket now, only have to answer reviewers comments.</p> <p>I have also observed this on a filesystem created with 2.1.x and migrated to 2.4.2.<br/> Exactly the same error message but this happens only with some directories.<br/> Some directories lfs_migrate'd perfectly.</p> <p>backport to b2_5:<br/> <a href="http://review.whamcloud.com/9278" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9278</a></p> <p>Landed for 2.5.1 and 2.6</p> Related LU-2445 LU-3826 LU-3696 Development Epic lfs_migrate Rank 1|hzw9x3: Rank (Obsolete) 11777 Severity