https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3190 Lustre <p>Hit following LBUG when running lustre-rsync-test test 3b<br/> In tag-2.3.62, the same test passed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == lustre-rsync-test test 3b: Replicate files created by writemany == 17:57:47 (1366246667) LustreError: 6661:0:(lmv_obd.c:850:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd c0086696: err = -2 LustreError: 6661:0:(lmv_obd.c:850:lmv_iocontrol()) Skipped 1415 previous similar messages Lustre: DEBUG MARKER: == lustre-rsync-test test 3c: Replicate files created by createmany/unlinkmany == 17:59:17 (1366246757) Lustre: DEBUG MARKER: == lustre-rsync-test test 4: Replicate files created by iozone == 17:59:33 (1366246773) LustreError: 7489:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object [0x20001d0f0:0x340d:0x0]: -95 LustreError: 7489:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object [0x20001d0f0:0x340d:0x0]: -95 LustreError: 7489:0:(lov_io.c:311:lov_io_slice_init()) ASSERTION( lio-&gt;lis_lsm != ((void *)0) ) failed: LustreError: 7489:0:(lov_io.c:311:lov_io_slice_init()) LBUG Pid: 7489, comm: lustre_rsync Message from Call Trace: syslogd@client- [&lt;ffffffffa0996905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 5 at Apr 17 18:0 [&lt;ffffffffa0996f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 0:04 ... kern [&lt;ffffffffa06b5088&gt;] lov_io_init_raid0+0x6d8/0x810 [lov] el:LustreError: [&lt;ffffffffa06ac037&gt;] lov_io_init+0x97/0x160 [lov] 7489:0:(lov_io.c [&lt;ffffffffa0dd1578&gt;] cl_io_init0+0x98/0x160 [obdclass] [&lt;ffffffffa0dd4464&gt;] cl_io_init+0x64/0x100 [obdclass] [&lt;ffffffffa07e6fed&gt;] cl_glimpse_size0+0x7d/0x190 [lustre] :311:lov_io_slic [&lt;ffffffffa07a3f32&gt;] ll_inode_revalidate_it+0xf2/0x1c0 [lustre] [&lt;ffffffffa07a4049&gt;] ll_getattr_it+0x49/0x170 [lustre] [&lt;ffffffffa07a41a7&gt;] ll_getattr+0x37/0x40 [lustre] [&lt;ffffffff81214343&gt;] ? security_inode_getattr+0x23/0x30 e_init()) ASSERT [&lt;ffffffff81180571&gt;] vfs_getattr+0x51/0x80 [&lt;ffffffffa09a2088&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffff8118082f&gt;] vfs_fstat+0x3f/0x60 [&lt;ffffffff81180874&gt;] sys_newfstat+0x24/0x40 [&lt;ffffffff810d6b12&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b ION( lio-&gt;lis_lsKernel panic - not syncing: LBUG m != ((void *)0)Pid: 7489, comm: lustre_rsync Not tainted 2.6.32-279.5.1.el6.x86_64 #1 ) failed: Call Trace: Message from s [&lt;ffffffff814fd24a&gt;] ? panic+0xa0/0x168 yslogd@client-5 [&lt;ffffffffa0996f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] at Apr 17 18:00: [&lt;ffffffffa06b5088&gt;] ? lov_io_init_raid0+0x6d8/0x810 [lov] 04 ... kernel [&lt;ffffffffa06ac037&gt;] ? lov_io_init+0x97/0x160 [lov] :LustreError: 74 [&lt;ffffffffa0dd1578&gt;] ? cl_io_init0+0x98/0x160 [obdclass] 89:0:(lov_io.c:3 [&lt;ffffffffa0dd4464&gt;] ? cl_io_init+0x64/0x100 [obdclass] 11:lov_io_slice_ [&lt;ffffffffa07e6fed&gt;] ? cl_glimpse_size0+0x7d/0x190 [lustre] init()) LBUG [&lt;ffffffffa07a3f32&gt;] ? ll_inode_revalidate_it+0xf2/0x1c0 [lustre] Message from [&lt;ffffffffa07a4049&gt;] ? ll_getattr_it+0x49/0x170 [lustre] syslogd@client-5 [&lt;ffffffffa07a41a7&gt;] ? ll_getattr+0x37/0x40 [lustre] at Apr 17 18:00 [&lt;ffffffff81214343&gt;] ? security_inode_getattr+0x23/0x30 :04 ... kerne [&lt;ffffffff81180571&gt;] ? vfs_getattr+0x51/0x80 l:Kernel panic - [&lt;ffffffffa09a2088&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] not syncing: LB [&lt;ffffffff8118082f&gt;] ? vfs_fstat+0x3f/0x60 UG [&lt;ffffffff81180874&gt;] ? sys_newfstat+0x24/0x40 [&lt;ffffffff810d6b12&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff8100b0f2&gt;] ? system_call_fastpath+0x16/0x1b Initializing cgroup subsys cpuset </pre> </div></div> server: lustre-master tag-2.3.64 build #1411<br/> client: 2.3.0 LU-3190

Interop 2.3.0<->2.4 Failed on lustre-rsync-test test 3b: ASSERTION( lio->lis_lsm != ((void *)0) ) failed

Bug Major Resolved Fixed Zhenyu Xu Sarah Liu LB Thu, 18 Apr 2013 01:08:27 +0000 Mon, 1 Dec 2014 10:23:13 +0000 Mon, 1 Dec 2014 10:23:13 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.4.0 0 10 <p>Bobijam</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Sarah,</p> <p>Is it possible to collect some debug log? I've tried to reproduce the issue, but didn't make it.</p> <p>Important part here is to understand how did layout disappear and how likely it is to hit this in workloads other than lustre-rsync.</p> <p>I will try to reproduce it and get the debug log</p> <p>is it possible for the layout to be changed during the test which I didn't see in the test?</p> <p>However, the old clients have problem to handle layout change anyway. Should we block all IO by setting layout to empty, if a file's layout has been changed in an old client?</p> <p>I can reproduce this issue, it isn't only triggered by test_3b, just run the whole lustre-rsync-test and will hit it. The attached is debug log, please check.</p> <p>Will we release further 2.3.x lustre? The latest commit date of b2_3 is "Sat Oct 20 09:29:08 2012 -0400".</p> <p>Sarah,</p> <p>Could you please try this b2_3 patch when build finishes?</p> <p><a href="http://review.whamcloud.com/6167" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6167</a></p> <p>the test passed with the b2_3 patch<br/> <a href="https://maloo.whamcloud.com/test_sets/3566a5f4-ae23-11e2-bbea-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3566a5f4-ae23-11e2-bbea-52540035b04c</a></p> <p>It looks like we can actually address this issue on master to guard old clients from tripping.</p> <p>2..0 doe not set OBD_CONNECT_LAYOUTLOCK so the feature is supposed to be disabled for such clients. Yet we apparently still grant layout lock back to them?<br/> it's ok for 2.1 clients, they would just ignore unknown bits, but 2.3 clients actually know about hte bit.<br/> So the solution is probably to mask off unknown bits server-side when forming RPCs, I think.</p> <p>2.3 clients just have that bit defined but not enabled. The server didn't grant layout lock back. Just a different layout was granted back and client had problem to handle it. Actually I don't know why the layout was changed - it'd be great if we can collect a log about this. 2.3 client did report an error however the error was wrongly ignored.</p> <p>Do we do compatibility test between 2.1/2.2 and 2.4 servers? </p> <p>Jinshan</p> <p>We do test 2.1.x clients with 2.4 servers but not 2.2</p> <p>Peter</p> <p>Jinshan: different layout granted back during rsync does not make much sense, unless the transition was from NULL to populated layout, but this is actually a long-supported layout change, so should be fine.</p> <p>There's a debug log attached to this ticket, if it's not enough, can you please describe what additional stuff would you like to see there?</p> <p>Jinshan is now taking the test nodes which I used to reproduce this issue. We also found with the latest master build #1446, the lustre-rsync-test failed in the test_1 as "FAIL: Failure in replication; differences found." between 2.3 client and 2.4 server, I will file a separated ticket when I get the nodes back. </p> <p>I did some investigation on this issue, I guess FIDs on the MDT is wrong.</p> <p>In the rsync test, it creates link files and uses fid2path for replication. However, fid2path is disordered as the following can happen:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-5 ~]# lfs getstripe /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3394 0xd42 0 [root@client-5 ~]# lfs getstripe /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 3394 0xd42 0 [root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 [0x200000402:0x38:0x0] [root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 [0x200000402:0x38:0x0] [root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000402:0x29:0x0]' /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 [root@client-5 ~]# lfs fid2path /mnt/lustre "[0x200000402:0x38:0x0]" /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 </pre> </div></div> <p>link1 and link2 are hard link and the FID is '<span class="error">&#91;0x200000402:0x38:0x0&#93;</span>'.</p> <p>FID '<span class="error">&#91;0x200000402:0x29:0x0&#93;</span>' is the file which it hit LBUG. I also dumped the layout unfortunately the objid is not 0xd42.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LustreError: 30192:0:(lov_object.c:641:lov_conf_set()) lov ffff880329d78b58 changing lsm ffff8802dcae81c0 to ffff88031e69c3c0 failed LustreError: 30192:0:(debug.c:70:dump_lsm()) lsm ffff8802dcae81c0, objid 0x29, maxbytes 0xffffffff000, magic 0x0BD10BD0, stripe_size 1048576, stripe_count 1, refc: 1, layout_gen 0, pool [] LustreError: 30192:0:(debug.c:70:dump_lsm()) lsm ffff88031e69c3c0, objid 0x38, maxbytes 0xffffffff000, magic 0x0BD10BD0, stripe_size 1048576, stripe_count 1, refc: 1, layout_gen 0, pool [] LustreError: 30192:0:(lcommon_cl.c:1210:cl_file_inode_init()) Failure to initialize cl object ffff8802dc6dfa48 [0x200000402:0x29:0x0]: -95 </pre> </div></div> <p>The object id are 0x29 and 0x38 respectively. I guess somehow the fid in the dir entry for hard link is broken.</p> <p>Also in the test, I've ever seen that two FID2 were pointing to the same file. There is no opening file in the system at that time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000400:0x1a0d:0x0]' /mnt/lustre/d0.lustre-rsync-test/d2/clients/client0/~dmtmp/ACCESS/INV.PRN [root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000400:0x1a0b:0x0]' /mnt/lustre/d0.lustre-rsync-test/d2/clients/client0/~dmtmp/ACCESS/INV.PRN </pre> </div></div> <p>I guess there is some problem with fid2path code. I will investigate this.</p> <p>Apparently the same file would have different FIDs if it's being accessed from directory hierarchy and fid2path ways. I don't think this is a problem about client IO code.</p> <p>I think there are something wrong with changelog and linkea code. Unfortunately I'm not an expert about this piece of code.</p> <p>Di, you were recently working on the fid2path code for DNE in <a href="http://review.whamcloud.com/5676" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5676</a>. Is it possible this patch introduced a defect?</p> <p>Hmm, it seems this bug only exists between 2.3 clients and 2.4 server, i.e. it works fine with master. Patch 5676 mostly are server side changes, there are some changes on the client side, but all about DNE. So I would say unlikely, but I will investigate it a bit.</p> <p>It looks like stale FIDs can still exist in the linkea or OI table. This confused the client because sometimes two FIDs claimed the same OST objects and sometimes a file's stripe data can change. However, this can only happen for changelog and fid2path code.</p> <p>Also, from the snippet of mdt_getattr_internal() in mdt_handler.c,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (mdt_body_has_lov(la, reqbody)) { <span class="code-keyword">if</span> (ma-&gt;ma_valid &amp; MA_LOV) { LASSERT(ma-&gt;ma_lmm_size); mdt_dump_lmm(D_INFO, ma-&gt;ma_lmm); </pre> </div></div> <p>From the log message, the FID in layout dumped here is not in sync with MDT object.</p> <p>Is there some idea why the FID became stale in the first place? Is this rename-over-open-file or similar? In that case, it seems possible that a FID (inode) still exists and has the same pathname as another FID. In the kernel, when this happens to files in /proc/</p> {pid} <p>/fd/* they append " (deleted)" at the end of the pathname to make this obvious...</p> <p>Another thing interesting(according to jinshan's comment on 30/Apr/13 4:51 AM) is that, Changelog somehow provide <span class="error">&#91;0x200000402:0x29:0x0&#93;</span>, then by .lustre, it retrieve the inode with FID <span class="error">&#91;0x200000402:0x38:0x0&#93;</span>, which can explain why fid2path return the same path for different FID. And also according to the comments </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 [0x200000402:0x38:0x0] [root@client-5 ~]# lfs path2fid /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 [0x200000402:0x38:0x0] [root@client-5 ~]# lfs fid2path /mnt/lustre '[0x200000402:0x29:0x0]' /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 [root@client-5 ~]# lfs fid2path /mnt/lustre "[0x200000402:0x38:0x0]" /mnt/lustre/d0.lustre-rsync-test/d1/d1/link1 /mnt/lustre/d0.lustre-rsync-test/d1/d1/link2 </pre> </div></div> <p>According to the test script in lustre-rsync-test.sh</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #hard and soft links cat /etc/hosts &gt; $DIR/$tdir/d1/link1 ln $DIR/$tdir/d1/link1 $DIR/$tdir/d1/link2 ln -s $DIR/$tdir/d1/link1 $DIR/$tdir/d1/link3 </pre> </div></div> <p>That is the only related operation for link1 and link2, it seems unlikely changelog will mess up here. So I suspect OI table might be messed up somehow, but this can not explain why<br/> 1. only 2.3 clients trigger this problem.<br/> 2. where does this <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> come from?</p> <p>Fang yong, Could you please comment? Thanks. </p> <p>Why do you think the <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> and <span class="error">&#91;0x200000402:0x38:0x0&#93;</span> map to the same inode? Is it possible that the <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> was the FID of old link1 for former test_1? And for some reason, the old link1's inode was not destroyed. Then 'fid2path' on <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> will show the same result as <span class="error">&#91;0x200000402:0x38:0x0&#93;</span> does.</p> <p>According to the test_1 create history, the FID <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> does not belong to current test_1. To verify that, please try on current link1:</p> <p>1) ln $DIR/$tdir/d1/link1 $DIR/$tdir/d1/foo</p> <p>2) lfs fid2path /mnt/lustre '<span class="error">&#91;0x200000402:0x29:0x0&#93;</span>'</p> <p>If the result does not show '/mnt/lustre/d0.lustre-rsync-test/d1/d1/foo', then <span class="error">&#91;0x200000402:0x29:0x0&#93;</span> and <span class="error">&#91;0x200000402:0x38:0x0&#93;</span> map to the different inode.</p> <p>On the other hand, please show "/proc/fs/lustre/osd-ldiskfs/lustre-MDT0000/oi_scrub" and "/proc/fs/lustre/mdd/lustre-MDT0000/lfsck_namespace" when hit the issues again. Thanks!</p> <p>Is it possible that the bug here is that the inode's linkEA was not removed when the unlink was done?</p> <p>It seems to me that the '<span class="error">&#91;0x200000402:0x29:0x0&#93;</span>' FID shouldn't have a linkEA that still has the "link1" and "link2" names in it for that parent directory, if it has been deleted. At that point fid2path should just return nothing for an open-unlinked file.</p> <p>Alternately, is it possible that we are not deleting the unlinked file from the OI, and it is finding the FID-&gt;inode mapping for a deleted file? I see the following code which may be causing this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mdd_unlink()-&gt; mdd_finish_unlink() { [AD] The <span class="code-quote">"rc == 0"</span> check here is completely redundant and should be removed <span class="code-keyword">if</span> (rc == 0 &amp;&amp; (ma-&gt;ma_attr.la_nlink == 0 || is_dir)) obj-&gt;mod_flags |= DEAD_OBJ; } mdd_links_del()-&gt; mdd_links_rename()-&gt; <span class="code-object">int</span> mdd_linkea_prepare() { <span class="code-keyword">if</span> (mdd_obj-&gt;mod_flags &amp; DEAD_OBJ) <span class="code-comment">/* No more links, don't bother */</span> RETURN(0); } </pre> </div></div> <p>So this is marking the object with DEAD_OBJ when it has nlink == 0, regardless of whether it is still open or not, and this skips removal of the linkEA entries. If this object can still be found via fid2path() then it will return incorrect pathnames. If the file is deleted, then it shouldn't be accessible via fid2path() at all, just returning -ENOENT. If the file is open-unlinked, then the linkEA removal should not be skipped, so fid2path() returns nothing at all.</p> <p>According to xiong's comment(30/Apr/13 10:39 PM), the object we got by FID, has different lmm_oi in its EA, i.e. client try to get object by <span class="error">&#91;0x200000402:0x29:0x0&#93;</span>, but get one object with <span class="error">&#91;0x200000402:0x38:0x0&#93;</span> in its lmm_oi. That is why I suspect OI return me wrong object. Otherwise, FIDs and lmm_oi are not in sync, which seems unlikely to me. But I do not have further proof for now. Sigh, I can not reproduce this ticket locally on my VM.</p> <p>After discussing this with Jinshan and WangDi, the issue at hand is mostly about accessing stuff via .lustre/fid interface, so seems to be pretty minor for the interop scenario.</p> <p>The dangerous part is the stale fid access might still be there in 2.4 as well (and seems to be a recent addition too), it's just less visible because 2.4 does not panic on layout change.</p> <p>Still overall I thing this does not need to be such a high blocker anymore.</p> <p>I just checked the debug log carefully, it seems OI cache is fine. Sorry about the noise. The real issue is that even dead object(deleted from the namespace) will still return linkEA, this is wrong. (So Andreas's comment on 02/May/13 6:37 PM) is correct. I will cook a patch.</p> <p><a href="http://review.whamcloud.com/#change,6252" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6252</a></p> <p>Suddenly I realize the current linkea is okay. There is nothing wrong for two FIDs to point to the same file path - actually they are two separate files. Sorry for misleading.</p> <p>So the problem boils down to the inconsistent layout was be returned through OBF, as what I mentioned at comment "30/Apr/13 3:39 PM". Is there any CLI tool to map a FID to the inode # in the OSD?</p> <p>It turns out OI cache problem, as jinshan said, object FID is different with the lmm_oi, so I add this patch in osd_xattr_get to dump some information.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- return __osd_xattr_get(inode, dentry, name, buf-&gt;lb_buf, buf-&gt;lb_len); + rc = __osd_xattr_get(inode, dentry, name, buf-&gt;lb_buf, buf-&gt;lb_len); + if (strcmp(name, XATTR_NAME_LOV) == 0 &amp;&amp; rc &gt; 0) { + struct lov_mds_md *md = (struct lov_mds_md *)buf-&gt;lb_buf; + if (unlikely(fid_oid(lu_object_fid(&amp;dt-&gt;do_lu)) != md-&gt;lmm_oi.oi.oi_id)) { + struct lu_fid fid = {0}; + struct osd_object *lmm_oi_obj; + struct osd_inode_id id1 = {0}, id2 = {0}, id3 = {0}; + struct osd_device *osd = osd_obj2dev(obj); + int rc1,rc2, rc0; + + fid.f_oid = md-&gt;lmm_oi.oi.oi_id; + fid.f_seq = md-&gt;lmm_oi.oi.oi_seq; + CERROR("Uncorrect layout for "DFID" FID is "DFID" "DFID" %u:%u\n", + PFID(lu_object_fid(&amp;dt-&gt;do_lu)), PFID(&amp;fid), PFID(&amp;info-&gt;oti_cache.oic_fid), info-&gt;oti_cache.oic_lid.oii_ino, info-&gt;oti_cache.oic_lid.oii_gen); + + rc0 = osd_oii_lookup(osd, lu_object_fid(&amp;dt-&gt;do_lu), &amp;id3); + rc1 = osd_oi_lookup(info, osd, &amp;fid, &amp;id1, false); + rc2 = osd_oi_lookup(info, osd, lu_object_fid(&amp;dt-&gt;do_lu), &amp;id2, false); + + lmm_oi_obj = osd_object_find(env, dt, &amp;fid); + if(IS_ERR(lmm_oi_obj)) { + CERROR("can not find obj by "DFID"\n", PFID(&amp;fid)); + } else { + CERROR("object inode %p %p %lu %u:%u rc %d dt obj %p %p %lu %u:%u Rc %d rc0 %d id3 %u:%u\n", lmm_oi_obj-&gt;oo_inode, lmm_oi_obj-&gt;oo_inode, lmm_oi_obj-&gt;oo_inode-&gt;i_ino, id1.oii_ino, id1.oii_gen, rc1, + obj, obj-&gt;oo_inode, obj-&gt;oo_inode-&gt;i_ino, id2.oii_ino, id2.oii_gen, rc2, rc0, id3.oii_ino, id3.oii_gen); + } + + LBUG(); + } + } + return rc; } </pre> </div></div> <p>And when the error happens, the output are</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == lustre-rsync-test test 4: Replicate files created by iozone == 01:15:24 (1367655324) LustreError: 3606:0:(osd_handler.c:2647:osd_xattr_get()) Uncorrect layout for [0x200000400:0x586f:0x0] FID is [0x200000400:0x5871:0x0] [0x200000400:0x586f:0x0] 161:818259173 (oic_cache) LustreError: 3606:0:(osd_handler.c:2658:osd_xattr_get()) object inode ffff88007b5a7ba0 ffff88007b5a7ba0 161 161:818259173 rc 0 dt obj ffff880065a8e818 ffff88007b5a7ba0 161 0:0 Rc -2 rc0 -2 id3 0:0 LustreError: 3606:0:(osd_handler.c:2661:osd_xattr_get()) LBUG Pid: 3606, comm: mdt00_002 </pre> </div></div> <p>So we can see both <span class="error">&#91;0x200000400:0x586f:0x0&#93;</span> (unlinked one) and <span class="error">&#91;0x200000400:0x5871:0x0&#93;</span>(corrected one) are pointing to the inode ffff88007b5a7ba0(161), but only correct one can be found in osd_oi_lookup, which means OI table is correct. But it seems the unlinked pair is left in the oi cache(<span class="error">&#91;0x200000400:0x586f:0x0&#93;</span>, 161), so we can still get one inode 161 from osd_fid_lookup. But unfortunately inode 161 has been be used to by the new object(<span class="error">&#91;0x200000400:0x5871:0x0&#93;</span>), that is why we different lmm_oi here. And it seems this patch are good enough to fix the problem</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/osd-ldiskfs/osd_oi.c b/lustre/osd-ldiskfs/osd_oi.c index 574b706..dc7f5e8 100644 --- a/lustre/osd-ldiskfs/osd_oi.c +++ b/lustre/osd-ldiskfs/osd_oi.c @@ -674,6 +674,10 @@ int osd_oi_delete(struct osd_thread_info *info, { struct lu_fid *oi_fid = &amp;info-&gt;oti_fid2; + /* clear idmap cache */ + if (lu_fid_eq(fid, &amp;info-&gt;oti_cache.oic_fid)) + memset(&amp;info-&gt;oti_cache, 0, sizeof(struct osd_idmap_cache)); + if (fid_is_last_id(fid)) return 0; </pre> </div></div> <p>Jinshan,<br/> I don't agree with your comment that it is OK to have two inodes with the same pathname at the same time. That isn't possible in the namespace, just like you cannot have two files in the same directory with the same filename at the same time. I'm not objecting to two inodes that had the same name at different times (which seems to be the case here), but only the new one should resolve to the pathname with fid2path(). The old file can return any <em>other</em> pathnames that it still has, or return an ENOENT error if it is unlinked.</p> <p>Di,<br/> it still seems to make sense to not return any pathnames in the case of an open-unlinked inode. In this case, even if the oti_cache was pointing to the unlinked inode, there shouldn't have been any entries in the "link" xattr to return, so either the inode didn't get written to disk after it was unlinked, or the "link" entries are not being written for unlinked inodes. I still think that case needs to be handled properly, and checking the "DEAD" and "ORPHAN" flags is probably the right way to go.</p> <p>Yes, I totally agree we should not return LinkEA for the object once it is removed from the namespace.</p> <p>After running patch set 5 for 3 times, cannot reproduce this issue, usually will hit this bug before sub test_5. All runs failed on sub test_7, please refer to <a href="https://jira.whamcloud.com/browse/LU-3279" title="Interop 2.3.0 &lt;-&gt; 2.4 failure on test suite lustre-rsync-test test_7: Failure in replication; differences found" class="issue-link" data-issue-key="LU-3279"><del>LU-3279</del></a></p> <p><a href="https://maloo.whamcloud.com/test_sets/5584e96e-b5de-11e2-9d08-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5584e96e-b5de-11e2-9d08-52540035b04c</a></p> <p>Sigh, only remove oi_cache in current thread info seems not enough, and we should remove this oi in the cache of all thread infos. </p> <p>In fact, the per-thread based "FID =&gt; ino#/gen" mapping should be fixed, if someone removed the inode, and caused the OI mapping deleted from the OI file, the related "inode::n_link" should be zero, or if the inode is reused by others, then the "inode::i_generation" will be changed. So other threads cached old "FID =&gt; ion/gen" mapping will be invalid automatically, because nobody can use the old "ino/gen" to find out the inode. One special case is that: if the cached generation is "OSD_OII_NOGEN", then we need verify with the inode's LMA.</p> <p>hmm, we probably should not put OSD_OII_NOGEN to the oi_cache, which make us unable to compare the generation at all. Just update the patch, please have a look.</p> <p>Patch set 7 didn't hit the LBUG but still failed test_7 as <a href="https://jira.whamcloud.com/browse/LU-3279" title="Interop 2.3.0 &lt;-&gt; 2.4 failure on test suite lustre-rsync-test test_7: Failure in replication; differences found" class="issue-link" data-issue-key="LU-3279"><del>LU-3279</del></a></p> <p>Lower priority as patch landed to master.<br/> But <a href="http://review.whamcloud.com/#change,6167" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6167</a> still needs to land.</p> <p>With the <a href="http://review.whamcloud.com/6252" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6252</a> patch landed to return -ENOENT for old FIDs (or is that -ESTALE?), does lustre-rsync-test still crash on a 2.3 client? Is change 6167 even needed for b2_3 anymore?</p> <p>Fixed for 2.4.0, won't fix for 2.3.0 anymore.</p> Related LU-3279 Development Rank 1|hzvogf: Rank (Obsolete) 7783 Severity