[LU-3190] Interop 2.3.0<->2.4 Failed on lustre-rsync-test test 3b: ASSERTION( lio->lis_lsm != ((void *)0) ) failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.3.0, Lustre 2.4.0
• Fix Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Major
• Reporter: Sarah Liu
• Assignee: Zhenyu Xu
• Resolution: Fixed
• Labels: LB

Environment

server: lustre-master tag-2.3.64 build #1411
client: 2.3.0

• Severity: 3

Description

Hit following LBUG when running lustre-rsync-test test 3b
In tag-2.3.62, the same test passed

Lustre: DEBUG MARKER: == lustre-rsync-test test 3b: Replicate files created by writemany == 17:57:47 (1366246667)
LustreError: 6661:0:(lmv_obd.c:850:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd c0086696: err = -2
LustreError: 6661:0:(lmv_obd.c:850:lmv_iocontrol()) Skipped 1415 previous similar messages
Lustre: DEBUG MARKER: == lustre-rsync-test test 3c: Replicate files created by createmany/unlinkmany == 17:59:17 (1366246757)
Lustre: DEBUG MARKER: == lustre-rsync-test test 4: Replicate files created by iozone == 17:59:33 (1366246773)
LustreError: 7489:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object [0x20001d0f0:0x340d:0x0]: -95
LustreError: 7489:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object [0x20001d0f0:0x340d:0x0]: -95
LustreError: 7489:0:(lov_io.c:311:lov_io_slice_init()) ASSERTION( lio->lis_lsm != ((void *)0) ) failed: 
LustreError: 7489:0:(lov_io.c:311:lov_io_slice_init()) LBUG
Pid: 7489, comm: lustre_rsync

Message from
Call Trace:
 syslogd@client- [<ffffffffa0996905>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
5 at Apr 17 18:0 [<ffffffffa0996f17>] lbug_with_loc+0x47/0xb0 [libcfs]
0:04 ...
 kern [<ffffffffa06b5088>] lov_io_init_raid0+0x6d8/0x810 [lov]
el:LustreError:  [<ffffffffa06ac037>] lov_io_init+0x97/0x160 [lov]
7489:0:(lov_io.c [<ffffffffa0dd1578>] cl_io_init0+0x98/0x160 [obdclass]
 [<ffffffffa0dd4464>] cl_io_init+0x64/0x100 [obdclass]
 [<ffffffffa07e6fed>] cl_glimpse_size0+0x7d/0x190 [lustre]
:311:lov_io_slic [<ffffffffa07a3f32>] ll_inode_revalidate_it+0xf2/0x1c0 [lustre]
 [<ffffffffa07a4049>] ll_getattr_it+0x49/0x170 [lustre]
 [<ffffffffa07a41a7>] ll_getattr+0x37/0x40 [lustre]
 [<ffffffff81214343>] ? security_inode_getattr+0x23/0x30
e_init()) ASSERT [<ffffffff81180571>] vfs_getattr+0x51/0x80
 [<ffffffffa09a2088>] ? libcfs_log_return+0x28/0x40 [libcfs]
 [<ffffffff8118082f>] vfs_fstat+0x3f/0x60
 [<ffffffff81180874>] sys_newfstat+0x24/0x40
 [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b

ION( lio->lis_lsKernel panic - not syncing: LBUG
m != ((void *)0)Pid: 7489, comm: lustre_rsync Not tainted 2.6.32-279.5.1.el6.x86_64 #1
 ) failed: 
Call Trace:

Message from s [<ffffffff814fd24a>] ? panic+0xa0/0x168
yslogd@client-5  [<ffffffffa0996f6b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
at Apr 17 18:00: [<ffffffffa06b5088>] ? lov_io_init_raid0+0x6d8/0x810 [lov]
04 ...
 kernel [<ffffffffa06ac037>] ? lov_io_init+0x97/0x160 [lov]
:LustreError: 74 [<ffffffffa0dd1578>] ? cl_io_init0+0x98/0x160 [obdclass]
89:0:(lov_io.c:3 [<ffffffffa0dd4464>] ? cl_io_init+0x64/0x100 [obdclass]
11:lov_io_slice_ [<ffffffffa07e6fed>] ? cl_glimpse_size0+0x7d/0x190 [lustre]
init()) LBUG
 [<ffffffffa07a3f32>] ? ll_inode_revalidate_it+0xf2/0x1c0 [lustre]

Message from  [<ffffffffa07a4049>] ? ll_getattr_it+0x49/0x170 [lustre]
syslogd@client-5 [<ffffffffa07a41a7>] ? ll_getattr+0x37/0x40 [lustre]
 at Apr 17 18:00 [<ffffffff81214343>] ? security_inode_getattr+0x23/0x30
:04 ...
 kerne [<ffffffff81180571>] ? vfs_getattr+0x51/0x80
l:Kernel panic - [<ffffffffa09a2088>] ? libcfs_log_return+0x28/0x40 [libcfs]
 not syncing: LB [<ffffffff8118082f>] ? vfs_fstat+0x3f/0x60
UG
 [<ffffffff81180874>] ? sys_newfstat+0x24/0x40
 [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
Initializing cgroup subsys cpuset

Issue Links

• is related to: LU-3279 Interop 2.3.0 <-> 2.4 failure on test...

Comments

[Peter Jones]

Bobijam

Could you please comment on this one?

Thanks

Peter

[Zhenyu Xu]

Sarah,

Is it possible to collect some debug log? I've tried to reproduce the issue, but didn't make it.

[Oleg Drokin]

Important part here is to understand how did layout disappear and how likely it is to hit this in workloads other than lustre-rsync.

[Sarah Liu]

I will try to reproduce it and get the debug log

[Jinshan Xiong (Inactive)]

is it possible for the layout to be changed during the test which I didn't see in the test?

However, the old clients have problem to handle layout change anyway. Should we block all IO by setting layout to empty, if a file's layout has been changed in an old client?

[Sarah Liu]

I can reproduce this issue, it isn't only triggered by test_3b, just run the whole lustre-rsync-test and will hit it. The attached is debug log, please check.

[Zhenyu Xu]

Will we release further 2.3.x lustre? The latest commit date of b2_3 is "Sat Oct 20 09:29:08 2012 -0400".

[Zhenyu Xu]

Sarah,

Could you please try this b2_3 patch when build finishes?

[Peter Jones]

http://review.whamcloud.com/6167

[Sarah Liu]

the test passed with the b2_3 patch
https://maloo.whamcloud.com/test_sets/3566a5f4-ae23-11e2-bbea-52540035b04c

[Oleg Drokin]

It looks like we can actually address this issue on master to guard old clients from tripping.

2..0 doe not set OBD_CONNECT_LAYOUTLOCK so the feature is supposed to be disabled for such clients. Yet we apparently still grant layout lock back to them?
it's ok for 2.1 clients, they would just ignore unknown bits, but 2.3 clients actually know about hte bit.
So the solution is probably to mask off unknown bits server-side when forming RPCs, I think.

[Jinshan Xiong (Inactive)]

2.3 clients just have that bit defined but not enabled. The server didn't grant layout lock back. Just a different layout was granted back and client had problem to handle it. Actually I don't know why the layout was changed - it'd be great if we can collect a log about this. 2.3 client did report an error however the error was wrongly ignored.

Do we do compatibility test between 2.1/2.2 and 2.4 servers?

[Peter Jones]

Jinshan

We do test 2.1.x clients with 2.4 servers but not 2.2

Peter

[Oleg Drokin]

Jinshan: different layout granted back during rsync does not make much sense, unless the transition was from NULL to populated layout, but this is actually a long-supported layout change, so should be fine.

There's a debug log attached to this ticket, if it's not enough, can you please describe what additional stuff would you like to see there?

[Sarah Liu]

Jinshan is now taking the test nodes which I used to reproduce this issue. We also found with the latest master build #1446, the lustre-rsync-test failed in the test_1 as "FAIL: Failure in replication; differences found." between 2.3 client and 2.4 server, I will file a separated ticket when I get the nodes back.

[Jinshan Xiong (Inactive)]

I did some investigation on this issue, I guess FIDs on the MDT is wrong.

In the rsync test, it creates link files and uses fid2path for replication. However, fid2path is disordered as the following can happen:

[root@client-5 ~]# lfs getstripe /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  0
	obdidx		 objid		objid		 group
	     0	          3394	        0xd42	             0

[root@client-5 ~]# lfs getstripe /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  0
	obdidx		 objid		objid		 group
	     0	          3394	        0xd42	             0

[root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
[0x200000402:0x38:0x0]
[root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
[0x200000402:0x38:0x0]
[root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000402:0x29:0x0]'
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
[root@client-5 ~]# lfs fid2path /mnt/lustre "[0x200000402:0x38:0x0]"
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link2

link1 and link2 are hard link and the FID is '[0x200000402:0x38:0x0]'.

FID '[0x200000402:0x29:0x0]' is the file which it hit LBUG. I also dumped the layout unfortunately the objid is not 0xd42.

LustreError: 30192:0:(lov_object.c:641:lov_conf_set()) lov ffff880329d78b58 changing lsm ffff8802dcae81c0 to ffff88031e69c3c0 failed
LustreError: 30192:0:(debug.c:70:dump_lsm()) lsm ffff8802dcae81c0, objid 0x29, maxbytes 0xffffffff000, magic 0x0BD10BD0, stripe_size 1048576, stripe_count 1, refc: 1, layout_gen 0, pool []
LustreError: 30192:0:(debug.c:70:dump_lsm()) lsm ffff88031e69c3c0, objid 0x38, maxbytes 0xffffffff000, magic 0x0BD10BD0, stripe_size 1048576, stripe_count 1, refc: 1, layout_gen 0, pool []
LustreError: 30192:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object ffff8802dc6dfa48 [0x200000402:0x29:0x0]: -95

The object id are 0x29 and 0x38 respectively. I guess somehow the fid in the dir entry for hard link is broken.

Also in the test, I've ever seen that two FID2 were pointing to the same file. There is no opening file in the system at that time.

[root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000400:0x1a0d:0x0]'
/mnt/lustre/d0.lustre-rsync-test/d2/clients/client0/~dmtmp/ACCESS/INV.PRN
[root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000400:0x1a0b:0x0]'
/mnt/lustre/d0.lustre-rsync-test/d2/clients/client0/~dmtmp/ACCESS/INV.PRN

I guess there is some problem with fid2path code. I will investigate this.

[Jinshan Xiong (Inactive)]

Apparently the same file would have different FIDs if it's being accessed from directory hierarchy and fid2path ways. I don't think this is a problem about client IO code.

I think there are something wrong with changelog and linkea code. Unfortunately I'm not an expert about this piece of code.

[Andreas Dilger]

Di, you were recently working on the fid2path code for DNE in http://review.whamcloud.com/5676. Is it possible this patch introduced a defect?

[Di Wang]

Hmm, it seems this bug only exists between 2.3 clients and 2.4 server, i.e. it works fine with master. Patch 5676 mostly are server side changes, there are some changes on the client side, but all about DNE. So I would say unlikely, but I will investigate it a bit.

[Jinshan Xiong (Inactive)]

It looks like stale FIDs can still exist in the linkea or OI table. This confused the client because sometimes two FIDs claimed the same OST objects and sometimes a file's stripe data can change. However, this can only happen for changelog and fid2path code.

[Jinshan Xiong (Inactive)]

Also, from the snippet of mdt_getattr_internal() in mdt_handler.c,

        if (mdt_body_has_lov(la, reqbody)) {
                if (ma->ma_valid & MA_LOV) {
                        LASSERT(ma->ma_lmm_size);
                        mdt_dump_lmm(D_INFO, ma->ma_lmm);

From the log message, the FID in layout dumped here is not in sync with MDT object.

[Andreas Dilger]

Is there some idea why the FID became stale in the first place? Is this rename-over-open-file or similar? In that case, it seems possible that a FID (inode) still exists and has the same pathname as another FID. In the kernel, when this happens to files in /proc/

{pid}

/fd/* they append " (deleted)" at the end of the pathname to make this obvious...

[Di Wang]

Another thing interesting(according to jinshan's comment on 30/Apr/13 4:51 AM) is that, Changelog somehow provide [0x200000402:0x29:0x0], then by .lustre, it retrieve the inode with FID [0x200000402:0x38:0x0], which can explain why fid2path return the same path for different FID. And also according to the comments

[root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
[0x200000402:0x38:0x0]
[root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
[0x200000402:0x38:0x0]
[root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000402:0x29:0x0]'
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link2
[root@client-5 ~]# lfs fid2path /mnt/lustre "[0x200000402:0x38:0x0]"
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link1
/mnt/lustre/d0.lustre-rsync-test/d1/d1/link2

According to the test script in lustre-rsync-test.sh

  #hard and soft links
    cat /etc/hosts > $DIR/$tdir/d1/link1
    ln  $DIR/$tdir/d1/link1  $DIR/$tdir/d1/link2
    ln -s $DIR/$tdir/d1/link1  $DIR/$tdir/d1/link3

That is the only related operation for link1 and link2, it seems unlikely changelog will mess up here. So I suspect OI table might be messed up somehow, but this can not explain why
1. only 2.3 clients trigger this problem.
2. where does this [0x200000402:0x29:0x0] come from?

Fang yong, Could you please comment? Thanks.

[nasf (Inactive)]

Why do you think the [0x200000402:0x29:0x0] and [0x200000402:0x38:0x0] map to the same inode? Is it possible that the [0x200000402:0x29:0x0] was the FID of old link1 for former test_1? And for some reason, the old link1's inode was not destroyed. Then 'fid2path' on [0x200000402:0x29:0x0] will show the same result as [0x200000402:0x38:0x0] does.

According to the test_1 create history, the FID [0x200000402:0x29:0x0] does not belong to current test_1. To verify that, please try on current link1:

1) ln $DIR/$tdir/d1/link1 $DIR/$tdir/d1/foo

2) lfs fid2path /mnt/lustre '[0x200000402:0x29:0x0]'

If the result does not show '/mnt/lustre/d0.lustre-rsync-test/d1/d1/foo', then [0x200000402:0x29:0x0] and [0x200000402:0x38:0x0] map to the different inode.

On the other hand, please show "/proc/fs/lustre/osd-ldiskfs/lustre-MDT0000/oi_scrub" and "/proc/fs/lustre/mdd/lustre-MDT0000/lfsck_namespace" when hit the issues again. Thanks!

[Andreas Dilger]

Is it possible that the bug here is that the inode's linkEA was not removed when the unlink was done?

It seems to me that the '[0x200000402:0x29:0x0]' FID shouldn't have a linkEA that still has the "link1" and "link2" names in it for that parent directory, if it has been deleted. At that point fid2path should just return nothing for an open-unlinked file.

[Andreas Dilger]

Alternately, is it possible that we are not deleting the unlinked file from the OI, and it is finding the FID->inode mapping for a deleted file? I see the following code which may be causing this:

mdd_unlink()->
  mdd_finish_unlink()
  {
        [AD] The "rc == 0" check here is completely redundant and should be removed
        if (rc == 0 && (ma->ma_attr.la_nlink == 0 || is_dir))
                obj->mod_flags |= DEAD_OBJ;
  }
  mdd_links_del()->
    mdd_links_rename()->
      int mdd_linkea_prepare()
      {
          if (mdd_obj->mod_flags & DEAD_OBJ)
                /* No more links, don't bother */
                RETURN(0);
      }

So this is marking the object with DEAD_OBJ when it has nlink == 0, regardless of whether it is still open or not, and this skips removal of the linkEA entries. If this object can still be found via fid2path() then it will return incorrect pathnames. If the file is deleted, then it shouldn't be accessible via fid2path() at all, just returning -ENOENT. If the file is open-unlinked, then the linkEA removal should not be skipped, so fid2path() returns nothing at all.

[Di Wang]

According to xiong's comment(30/Apr/13 10:39 PM), the object we got by FID, has different lmm_oi in its EA, i.e. client try to get object by [0x200000402:0x29:0x0], but get one object with [0x200000402:0x38:0x0] in its lmm_oi. That is why I suspect OI return me wrong object. Otherwise, FIDs and lmm_oi are not in sync, which seems unlikely to me. But I do not have further proof for now. Sigh, I can not reproduce this ticket locally on my VM.

[Oleg Drokin]

After discussing this with Jinshan and WangDi, the issue at hand is mostly about accessing stuff via .lustre/fid interface, so seems to be pretty minor for the interop scenario.

The dangerous part is the stale fid access might still be there in 2.4 as well (and seems to be a recent addition too), it's just less visible because 2.4 does not panic on layout change.

Still overall I thing this does not need to be such a high blocker anymore.

[Di Wang]

I just checked the debug log carefully, it seems OI cache is fine. Sorry about the noise. The real issue is that even dead object(deleted from the namespace) will still return linkEA, this is wrong. (So Andreas's comment on 02/May/13 6:37 PM) is correct. I will cook a patch.

[Di Wang]

http://review.whamcloud.com/#change,6252

[Jinshan Xiong (Inactive)]

Suddenly I realize the current linkea is okay. There is nothing wrong for two FIDs to point to the same file path - actually they are two separate files. Sorry for misleading.

So the problem boils down to the inconsistent layout was be returned through OBF, as what I mentioned at comment "30/Apr/13 3:39 PM". Is there any CLI tool to map a FID to the inode # in the OSD?

[Di Wang]

It turns out OI cache problem, as jinshan said, object FID is different with the lmm_oi, so I add this patch in osd_xattr_get to dump some information.

-       return __osd_xattr_get(inode, dentry, name, buf->lb_buf, buf->lb_len);
+       rc = __osd_xattr_get(inode, dentry, name, buf->lb_buf, buf->lb_len);
+       if (strcmp(name, XATTR_NAME_LOV) == 0 && rc > 0) {
+               struct lov_mds_md *md = (struct lov_mds_md *)buf->lb_buf;
+               if (unlikely(fid_oid(lu_object_fid(&dt->do_lu)) != md->lmm_oi.oi.oi_id)) {
+                       struct lu_fid fid = {0};
+                       struct osd_object *lmm_oi_obj;
+                       struct osd_inode_id id1 = {0}, id2 = {0}, id3 = {0};
+                       struct osd_device *osd = osd_obj2dev(obj);
+                       int rc1,rc2, rc0;
+
+                       fid.f_oid = md->lmm_oi.oi.oi_id;
+                       fid.f_seq = md->lmm_oi.oi.oi_seq;
+                       CERROR("Uncorrect layout for "DFID" FID is "DFID" "DFID" %u:%u\n",
+                              PFID(lu_object_fid(&dt->do_lu)), PFID(&fid), PFID(&info->oti_cache.oic_fid), info->oti_cache.oic_lid.oii_ino, info->oti_cache.oic_lid.oii_gen);
+
+                       rc0 = osd_oii_lookup(osd, lu_object_fid(&dt->do_lu), &id3);
+                       rc1 = osd_oi_lookup(info, osd, &fid, &id1, false);
+                       rc2 = osd_oi_lookup(info, osd, lu_object_fid(&dt->do_lu), &id2, false);
+
+                       lmm_oi_obj = osd_object_find(env, dt, &fid);
+                       if(IS_ERR(lmm_oi_obj)) {
+                               CERROR("can not find obj by "DFID"\n", PFID(&fid));
+                       } else {
+                               CERROR("object inode %p %p %lu %u:%u rc %d dt obj %p %p %lu %u:%u Rc %d rc0 %d id3 %u:%u\n", lmm_oi_obj->oo_inode, lmm_oi_obj->oo_inode, lmm_oi_obj->oo_inode->i_ino, id1.oii_ino, id1.oii_gen, rc1,
+                                       obj, obj->oo_inode, obj->oo_inode->i_ino, id2.oii_ino, id2.oii_gen, rc2, rc0, id3.oii_ino, id3.oii_gen); 
+                       }
+
+                       LBUG();
+               }
+       }
+       return rc;
 }

And when the error happens, the output are

Lustre: DEBUG MARKER: == lustre-rsync-test test 4: Replicate files created by iozone == 01:15:24 (1367655324)
LustreError: 3606:0:(osd_handler.c:2647:osd_xattr_get()) Uncorrect layout for [0x200000400:0x586f:0x0] FID is [0x200000400:0x5871:0x0] [0x200000400:0x586f:0x0] 161:818259173 (oic_cache)
LustreError: 3606:0:(osd_handler.c:2658:osd_xattr_get()) object inode ffff88007b5a7ba0 ffff88007b5a7ba0 161 161:818259173 rc 0 dt obj ffff880065a8e818 ffff88007b5a7ba0 161 0:0 Rc -2 rc0 -2 id3 0:0
LustreError: 3606:0:(osd_handler.c:2661:osd_xattr_get()) LBUG
Pid: 3606, comm: mdt00_002

So we can see both [0x200000400:0x586f:0x0] (unlinked one) and [0x200000400:0x5871:0x0](corrected one) are pointing to the inode ffff88007b5a7ba0(161), but only correct one can be found in osd_oi_lookup, which means OI table is correct. But it seems the unlinked pair is left in the oi cache([0x200000400:0x586f:0x0], 161), so we can still get one inode 161 from osd_fid_lookup. But unfortunately inode 161 has been be used to by the new object([0x200000400:0x5871:0x0]), that is why we different lmm_oi here. And it seems this patch are good enough to fix the problem

diff --git a/lustre/osd-ldiskfs/osd_oi.c b/lustre/osd-ldiskfs/osd_oi.c
index 574b706..dc7f5e8 100644
--- a/lustre/osd-ldiskfs/osd_oi.c
+++ b/lustre/osd-ldiskfs/osd_oi.c
@@ -674,6 +674,10 @@ int osd_oi_delete(struct osd_thread_info *info,
 {
        struct lu_fid *oi_fid = &info->oti_fid2;
 
+       /* clear idmap cache */
+       if (lu_fid_eq(fid, &info->oti_cache.oic_fid))
+               memset(&info->oti_cache, 0, sizeof(struct osd_idmap_cache));
+
        if (fid_is_last_id(fid))
                return 0;

[Andreas Dilger]

Jinshan,
I don't agree with your comment that it is OK to have two inodes with the same pathname at the same time. That isn't possible in the namespace, just like you cannot have two files in the same directory with the same filename at the same time. I'm not objecting to two inodes that had the same name at different times (which seems to be the case here), but only the new one should resolve to the pathname with fid2path(). The old file can return any other pathnames that it still has, or return an ENOENT error if it is unlinked.

Di,
it still seems to make sense to not return any pathnames in the case of an open-unlinked inode. In this case, even if the oti_cache was pointing to the unlinked inode, there shouldn't have been any entries in the "link" xattr to return, so either the inode didn't get written to disk after it was unlinked, or the "link" entries are not being written for unlinked inodes. I still think that case needs to be handled properly, and checking the "DEAD" and "ORPHAN" flags is probably the right way to go.

[Di Wang]

Yes, I totally agree we should not return LinkEA for the object once it is removed from the namespace.

[Sarah Liu]

After running patch set 5 for 3 times, cannot reproduce this issue, usually will hit this bug before sub test_5. All runs failed on sub test_7, please refer to LU-3279

https://maloo.whamcloud.com/test_sets/5584e96e-b5de-11e2-9d08-52540035b04c

[Di Wang]

Sigh, only remove oi_cache in current thread info seems not enough, and we should remove this oi in the cache of all thread infos.

[nasf (Inactive)]

In fact, the per-thread based "FID => ino#/gen" mapping should be fixed, if someone removed the inode, and caused the OI mapping deleted from the OI file, the related "inode::n_link" should be zero, or if the inode is reused by others, then the "inode::i_generation" will be changed. So other threads cached old "FID => ion/gen" mapping will be invalid automatically, because nobody can use the old "ino/gen" to find out the inode. One special case is that: if the cached generation is "OSD_OII_NOGEN", then we need verify with the inode's LMA.

[Di Wang]

hmm, we probably should not put OSD_OII_NOGEN to the oi_cache, which make us unable to compare the generation at all. Just update the patch, please have a look.

[Sarah Liu]

Patch set 7 didn't hit the LBUG but still failed test_7 as LU-3279

[Jodi Levi (Inactive)]

Lower priority as patch landed to master.
But http://review.whamcloud.com/#change,6167 still needs to land.

[Andreas Dilger]

With the http://review.whamcloud.com/6252 patch landed to return -ENOENT for old FIDs (or is that -ESTALE?), does lustre-rsync-test still crash on a 2.3 client? Is change 6167 even needed for b2_3 anymore?

[Andreas Dilger]

Fixed for 2.4.0, won't fix for 2.3.0 anymore.