|
Zam
Have you confirmed whether the situation improves with your proposed fix for LU-11130?
Peter
|
|
The bug still exists with patch from LU-11130.
However, we've found that the problem is no longer reproducible after disabling the cross-MDT capability provided with LU-3537 (but leaving the async update capability from LU-3534).
|
|
Peter,
>Have you confirmed whether the situation improves with your proposed fix for LU-11130
no, the unattached inodes have nothing to do with LU-11130. We have LU-11130 landed for some time already, it doesn't have any effect on this bug.
This issue is about regular files , racer.sh test (attached) without stiped dirs or migrate, no failovers, no e2fsck runs expept the final e2fsck run after racer.sh completes. Actually it was harder to reproduce the issue on master b/c it just hangs .
here the results of 10 min racer run:
E2fsck after racer. Well, it is mounted this time but nobody accesses the fs for about 30 sec since racer completes. Believe me the corruption exists when the fs is cleanly unmounted.
the inode stat:
FIDs in Link EA decoded:
Those fids found on the lustre fs:
Moreover, there is "8" file name in the second dir, but it doesn't allow file to be accessed:
in further investigation I found that fid lookup for [0x20001500f:0x5854:0x0] fails
(return code 0 means the fid not found):
Conclusions:
- the unattached inode on MDT0 actually has one link from MDT1 and it has a valid nlink counter value of 1
- the link from MDT1 doesn't work b/c FID lookup finds no inode on MDT0.
- the inode has no name on MDT0 which is a format violation, at least it should be connected to /REMOTE_PARENT_DIR/
- the OI on MDT0 has no record for [0x20001500f:0x5854:0x0] (the unattached inode FID)
- one of Link EA records is valid (corresponding forward link exists) another one is not vaild (there is no link "racer-dir/8").
|
|
More info from another run where I managed to get lustre logs with almost full debug (trace , inode, info, other) and I added debug code to OUT code :
the unattached inode FID is 0x20001ec49:0xab67:0x0 now. Below is a grep for 0x20001ec49:0xab67:0x0 through the lustre log:
For a moment, nlink counter dropped to zero:
I think it is not a valid situation, VFS warns about such cases, but Lustre avoids nlinks increase from zero (LU-4416 http://review.whamcloud.com/11571 ).
almost at the same time thread 61817 closes the file, it sees nlink = 0 (my guess) and continue with osd_delete_from_remote_parent() and osd_oi_delete() , it causes the corruption explained in the previous comment.
|
|
Attaching unconnected.txt.gz – lustre log from MDT0 with dropping nlink to zero and destroying the file.
|
|
more debugging showed the following scenario of the fs corruption:
1. client opens file [0x20001500b:0x185f4:0x0]/9 , FID=[0x200020b8a:0x1c3d:0x0] and keeps fd open:
2. rename [0x20001500b:0x185f4:0x0]/4 to [0x20001500b:0x185f4:0x0]/9 unlinks target file:
and puts the object into the orpan list b/c the file is still open:
3. somehow in parallel with (2), a cross-mdt transaction adds a reference to [0x200020b8a:0x1c3d:0x0] (increases nlink count):
it is a part of compound "out" update :
I think the cross-target update is a link() operation, creating a file name on another MDT.
4. finally, file close removes the object from orphan list and from OI (b/c it has ORPHAN_OBJ and DEAD_OBJ flags set), creating a fs corruption reported by e2fsck as "unattached inode".
I do not know what exactly is broken yet, suspecting cross-target link not checking properly for ORPAN_OBJ | DEAD_OBJ flags.
|
|
Alexander Zarochentsev (c17826@cray.com) uploaded a new patch: https://review.whamcloud.com/35021
Subject: LU-11549 mdt: take UPDATE lock for rename target
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4ff2700af396e63fd9ac90d5f51f1129465c8df7
|
|
Peter Jones - any chance we can get someone to review this?
|
|
Root cause update:
0. Locks are OK.
1. the mdt_reint_link operation over a remote object cannot perform a reliable check for the remote object, We want to check the mdd object for DEAD_OBJ flag but can't do that, because DEAD_OBJ is an mdd object flag, it exists on the remote MDT and not seen through OSP/OUT interface where out_attr_get() checks only dt object state. So the mdd_link_sanity_check() is passed successfully and illegal link operation is done.
2. A solution could be in moving DEAD_OBJ flag to lu_object level .
|
|
Alexander Zarochentsev (c17826@cray.com) uploaded a new patch: https://review.whamcloud.com/35683
Subject: LU-11549 tests: link succeded to an ophan remote object
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 14ab193c6aa78ca95b85fa5c940af62fbb815abb
|
|
Reproducer: https://review.whamcloud.com/35683 :
the wrong link causes a cross-mdt link corruption,
file2x is a name which cannot be deleted, it points to a remote object already removed from OI, but its inode (unconnected) still exists.
|
|
Alexander Zarochentsev (c17826@cray.com) uploaded a new patch: https://review.whamcloud.com/35776
Subject: LU-11549 mdd: set LUSTRE_ORPHAN_FL for non-dirs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fd759510de938a3556eddf3b5798ce95135e1dbc
|
|
The above patch https://review.whamcloud.com/35776 fixes the issue, my reproducer doesn't show any fs corruption . Unfortunately it reports an incorrect link creation even with the fix b/c the test was not ready that client repeats the link() operation after getting ESTALE error, so second attempt to create the link works.
the fix is one-liner:
diff --git a/lustre/mdd/mdd_dir.c b/lustre/mdd/mdd_dir.c
index 17613d6..260dd7a 100644
--- a/lustre/mdd/mdd_dir.c
+++ b/lustre/mdd/mdd_dir.c
@@ -1456,7 +1456,7 @@ static int mdd_mark_orphan_object(const struct lu_env *env,
struct lu_attr *attr = MDD_ENV_VAR(env, la_for_start);
int rc;
- if (!S_ISDIR(mdd_object_type(obj)))
+ if (S_ISDIR(mdd_object_type(obj)))
return 0;
attr->la_valid = LA_FLAGS;
|
|
Alexander Zarochentsev (c17826@cray.com) uploaded a new patch: https://review.whamcloud.com/35991
Subject: LU-11549 tests: link succeded to an ophan remote object
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3e6574997f82407c98b90318448400bb3a1ca5e0
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35776/
Subject: LU-11549 mdd: set LUSTRE_ORPHAN_FL for non-dirs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8d27c92a66d63aaf8b8fbe1fc73e49263b5bed1e
|
|
The fix for this problem has been landed for 2.13.0 so this ticket should be closed so that it can be tracked properly for the release. Since the test patch is not currently passing testing I've opened a separate ticket to track that landing.
|
Generated at Sat Feb 10 02:44:50 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
racer-mod.tar.gz