https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5163 Lustre <p>Running racer on 2.5.59-66-g47cde80 with MDSCOUNT=4 and file_create.sh modified to do fewer writes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2383.017816] LustreError: 16406:0:(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed: [ 2383.020138] LustreError: 16406:0:(lu_object.h:852:lu_object_attr()) LBUG [ 2383.021290] Pid: 16406, comm: mdt00_002 [ 2383.021909] [ 2383.021910] Call Trace: [ 2383.022612] [&lt;ffffffffa02be8c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 2383.023915] [&lt;ffffffffa02beec7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 2383.025200] [&lt;ffffffffa0bc6e8b&gt;] mdd_migrate_entries+0xffb/0x1100 [mdd] [ 2383.026503] [&lt;ffffffffa0bc7af7&gt;] mdd_migrate+0xb67/0x13d0 [mdd] [ 2383.027666] [&lt;ffffffffa0afc61b&gt;] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs] [ 2383.029154] [&lt;ffffffffa0c25c78&gt;] mdt_reint_migrate_internal+0x15c8/0x1b40 [mdt] [ 2383.030578] [&lt;ffffffff815547cb&gt;] ? _spin_unlock+0x2b/0x40 [ 2383.031524] [&lt;ffffffffa0c299d3&gt;] mdt_reint_rename_or_migrate+0x2a3/0x660 [mdt] [ 2383.032776] [&lt;ffffffffa0c03c95&gt;] ? mdt_ucred+0x15/0x20 [mdt] [ 2383.033739] [&lt;ffffffffa0c1e4cc&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] [ 2383.034849] [&lt;ffffffffa06b7026&gt;] ? __req_capsule_get+0x166/0x6e0 [ptlrpc] [ 2383.035986] [&lt;ffffffffa0c29da3&gt;] mdt_reint_migrate+0x13/0x20 [mdt] [ 2383.037045] [&lt;ffffffffa0c223e1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [ 2383.038083] [&lt;ffffffffa0c07e43&gt;] mdt_reint_internal+0x4c3/0x7c0 [mdt] [ 2383.039234] [&lt;ffffffffa0c086cb&gt;] mdt_reint+0x6b/0x120 [mdt] [ 2383.040228] [&lt;ffffffffa06f0a85&gt;] tgt_request_handle+0x245/0xad0 [ptlrpc] [ 2383.041371] [&lt;ffffffffa06a18d1&gt;] ptlrpc_main+0xce1/0x1970 [ptlrpc] [ 2383.042435] [&lt;ffffffffa06a0bf0&gt;] ? ptlrpc_main+0x0/0x1970 [ptlrpc] [ 2383.043468] [&lt;ffffffff8109eab6&gt;] kthread+0x96/0xa0 [ 2383.044261] [&lt;ffffffff8100c30a&gt;] child_rip+0xa/0x20 [ 2383.045066] [&lt;ffffffff81554710&gt;] ? _spin_unlock_irq+0x30/0x40 [ 2383.045986] [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [ 2383.046914] [&lt;ffffffff8109ea20&gt;] ? kthread+0x0/0xa0 [ 2383.047770] [&lt;ffffffff8100c300&gt;] ? child_rip+0x0/0x20 [ 2383.048666] [ 2383.049141] Kernel panic - not syncing: LBUG </pre> </div></div> <p>This is from mdd_migrate_entries() line 3473.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> child = mdd_object_find(env, mdd, &amp;ent-&gt;lde_fid); if (IS_ERR(child)) GOTO(out, rc = PTR_ERR(child)); is_dir = S_ISDIR(lu_object_attr(&amp;child-&gt;mod_obj.mo_lu)); </pre> </div></div> LU-5163

(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)->lo_header->loh_attr & LOHA_EXISTS) != 0 ) failed

Bug Blocker Resolved Fixed Lai Siyao John Hammond dne2 mdd migration Mon, 9 Jun 2014 14:25:47 +0000 Tue, 10 Jul 2018 14:43:47 +0000 Fri, 5 Jan 2018 13:42:31 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.11.0 Lustre 2.10.4 0 17 <p>Does this have the patch from <a href="https://jira.whamcloud.com/browse/LU-5069" title="Hit LBUG in DNE racer test: (lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-5069"><del>LU-5069</del></a> or is it a new issue?</p> <p>This is on today's master which has that patch. This is a new issue which is not related to rename sanity checking.</p> <p>Hmm, it seems the name entry is a dangling entry, and we probably can skip this name entry during migration. But I am not sure how this dangling entry is created, and not sure it is related with migration or not.</p> <p>Hmm, I did a few test, it seems this related with <a href="http://review.whamcloud.com/9538" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9538</a>, where it does is_subdir check without ldlm lock protection. Though I am not so sure, since it is not easy to get debug log to analyze it. But I tried to re-write the patch, use lookup(..), instead of is_subdir, it seems fix the LBUG issue, at least I can not reproduce the lbug with the patch <a href="http://review.whamcloud.com/10673" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10673</a></p> <p>I cherry-picked <a href="http://review.whamcloud.com/10673" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10673</a> onto today's master (2.5.59-79-g5c4573e) and I see the same LBUG with the same stack trace.</p> <p>Regardless of LDLM issues, this FID is coming straight from the disk so we should not be asserting on the existence of the object. I know that some may want to leave this in until they understand the underlying issue but I would prefer to have a less crashy implementation of an already best-effort type operation like migration.</p> <p>my take on this is that it's better to crash and restart instead of corrupting a filesystem silently (potentially, given we don't understand everything at the moment).</p> <p>Hit this again running racer on my single-node test system (2x MDT, 3x OST). The stack is a little different than the original one, so I thought I'd post it here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 25072:0:(lu_object.h:859:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed: LustreError: 25072:0:(lu_object.h:859:lu_object_attr()) LBUG Pid: 25072, comm: mdt00_005 Call Trace: [&lt;ffffffffa13a7895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa13a7e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa1015e07&gt;] lu_object_attr+0x47/0x50 [mdt] [&lt;ffffffffa10186b3&gt;] mdt_reint_migrate_internal+0x6a3/0x1b50 [mdt] [&lt;ffffffffa101d7cb&gt;] mdt_reint_rename_or_migrate+0x3cb/0x6c0 [mdt] [&lt;ffffffffa101dad3&gt;] mdt_reint_migrate+0x13/0x20 [mdt] [&lt;ffffffffa1015fcd&gt;] mdt_reint_rec+0x5d/0x200 [mdt] [&lt;ffffffffa0ffa19b&gt;] mdt_reint_internal+0x4cb/0x7a0 [mdt] [&lt;ffffffffa0ffa9fb&gt;] mdt_reint+0x6b/0x120 [mdt] [&lt;ffffffffa0c5398e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] [&lt;ffffffffa0c03721&gt;] ptlrpc_main+0xe41/0x1960 [ptlrpc] </pre> </div></div> <p>In particular, this is failing in mdt_reint_migrate_internal() rather than in mdd_migrate_entries():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(gdb) list *(mdt_reint_migrate_internal+0x6a3) 0x266e3 is in mdt_reint_migrate_internal (/usr/src/lustre-head/lustre/mdt/mdt_reint.c:1518). 1508 mold = mdt_object_find(info-&gt;mti_env, info-&gt;mti_mdt, old_fid); 1509 <span class="code-keyword">if</span> (IS_ERR(mold)) 1510 GOTO(out_unlock_parent, rc = PTR_ERR(mold)); 1511 1512 <span class="code-keyword">if</span> (mdt_object_remote(mold)) { 1513 CERROR(<span class="code-quote">"%s: source "</span>DFID<span class="code-quote">" is on the remote MDT\n"</span>, 1514 mdt_obd_name(info-&gt;mti_mdt), PFID(old_fid)); 1515 GOTO(out_put_child, rc = -EREMOTE); 1516 } 1517 1518 <span class="code-keyword">if</span> (S_ISREG(lu_object_attr(&amp;mold-&gt;mot_obj)) &amp;&amp; 1519 !mdt_object_remote(msrcdir)) { 1520 CERROR(<span class="code-quote">"%s: parent "</span>DFID<span class="code-quote">" is still on the same"</span> 1521 <span class="code-quote">" MDT, which should be migrated first:"</span> 1522 <span class="code-quote">" rc = %d\n"</span>, mdt_obd_name(info-&gt;mti_mdt), </pre> </div></div> <p>Is <tt>mold</tt> locked at this point after mdd_object_find-&gt;lu_object_find? Otherwise it seems entirely possible to delete the object between the time it is looked up and when the LASSERT() trips in lu_object_attr().</p> <p>Well, the parent has been locked, and the mold object is gotten by name-&gt;FID lookup. If the name entry exists, but the object (mold) does not exist, it means the name entry is becoming a dangling entry during the racer. </p> <p>this probably because we did not lock all of the children when migrating the directory. hmm</p> <p>I met this in local test, and am looking into it now.</p> <p>This looks to be in mdd_migrate_entries() -&gt; mdd_object_type(child), which asserts child should exist, while it's not always true, because child is not locked, I'll push a patch soon.</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26620" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26620</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5163" title="(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-5163"><del>LU-5163</del></a> mdd: obtain type from dirent<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c6fc00290924905e6c8c62bdecf283b04bc6f712</p> <p>Seen in master 2.10.55 b3667:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/0a9fe899-314e-4874-bd82-8a966bc7ad88" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0a9fe899-314e-4874-bd82-8a966bc7ad88</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/77756eb8-cb0a-11e7-9c63-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/77756eb8-cb0a-11e7-9c63-52540065bddc</a></p> <p>on 2.10.2<br/> <a href="https://testing.hpdd.intel.com/test_sets/c087a034-d4bb-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c087a034-d4bb-11e7-8027-52540065bddc</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26620/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26620/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5163" title="(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-5163"><del>LU-5163</del></a> mdd: migrated entry may not exist<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 542b76d142c491a0a1bf8a2f4fd22af4733f59cb</p> <p>Lai</p> <p>Can this ticket be marked as resolved now or is Di's patch still needed too?</p> <p>Peter</p> <p>Peter, it can be marked as resolved now, and Di's patch is not needed IMHO.</p> <p>ok - thanks!</p> <p>+1 on b2_10</p> <p><a href="https://testing.hpdd.intel.com/test_sets/cbf69ae0-0ed3-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cbf69ae0-0ed3-11e8-a6ad-52540065bddc</a></p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31281" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31281</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5163" title="(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-5163"><del>LU-5163</del></a> mdd: migrated entry may not exist<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: dac6aa7caf537d4d3e7a772c36fbdb355da47e56</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31281/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31281/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5163" title="(lu_object.h:852:lu_object_attr()) ASSERTION( ((o)-&gt;lo_header-&gt;loh_attr &amp; LOHA_EXISTS) != 0 ) failed" class="issue-link" data-issue-key="LU-5163"><del>LU-5163</del></a> mdd: migrated entry may not exist<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 8af27a326ffebc7c0f40734767035d353bafa43c</p> Duplicate LU-5242 Related LU-5125 LU-5388 LU-11135 Development Rank 1|hzwnyn: Rank (Obsolete) 14233 Severity