https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6996 Lustre <p>This morning a production MDS hit an assertion:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;[2551157.740086] LustreError: 14993:0:(osd_handler.c:4071:osd_ea_lookup_rec()) ASSERTION( dir-&gt;i_op != ((void *)0) &amp;&amp; dir-&gt;i_op-&gt;lookup != ((void *)0) ) failed: &lt;0&gt;[2551157.756253] LustreError: 14993:0:(osd_handler.c:4071:osd_ea_lookup_rec()) LBUG &lt;4&gt;[2551157.764766] Pid: 14993, comm: mdt01_094 &lt;4&gt;[2551157.769360] &lt;4&gt;[2551157.769361] Call Trace: &lt;4&gt;[2551157.774374] [&lt;ffffffffa0409895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[2551157.782474] [&lt;ffffffffa0409e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[2551157.789707] [&lt;ffffffffa0ca8fcf&gt;] osd_index_ea_lookup+0x6ff/0x8a0 [osd_ldiskfs] &lt;4&gt;[2551157.798308] [&lt;ffffffffa0d0dde0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] &lt;4&gt;[2551157.805733] [&lt;ffffffffa088c7c0&gt;] ? lod_index_lookup+0x0/0x30 [lod] &lt;4&gt;[2551157.813056] [&lt;ffffffffa088c7e5&gt;] lod_index_lookup+0x25/0x30 [lod] &lt;4&gt;[2551157.820291] [&lt;ffffffffa0dd0daa&gt;] __mdd_lookup+0x24a/0x440 [mdd] &lt;4&gt;[2551157.827325] [&lt;ffffffffa0dd1599&gt;] mdd_lookup+0x39/0xe0 [mdd] &lt;4&gt;[2551157.833977] [&lt;ffffffffa0d3bee5&gt;] ? mdt_name+0x35/0xc0 [mdt] &lt;4&gt;[2551157.840629] [&lt;ffffffffa0d44b09&gt;] mdt_reint_open+0xb69/0x21a0 [mdt] &lt;4&gt;[2551157.847959] [&lt;ffffffffa0426376&gt;] ? upcall_cache_get_entry+0x296/0x880 [libcfs] &lt;4&gt;[2551157.856570] [&lt;ffffffffa05c7a80&gt;] ? lu_ucred+0x20/0x30 [obdclass] &lt;4&gt;[2551157.863705] [&lt;ffffffffa0d2d481&gt;] mdt_reint_rec+0x41/0xe0 [mdt] &lt;4&gt;[2551157.870643] [&lt;ffffffffa0d12ed3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] &lt;4&gt;[2551157.878254] [&lt;ffffffffa0d1345e&gt;] mdt_intent_reint+0x1ee/0x410 [mdt] &lt;4&gt;[2551157.885669] [&lt;ffffffffa0d10c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] &lt;4&gt;[2551157.893212] [&lt;ffffffffa06e42e5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] &lt;4&gt;[2551157.901044] [&lt;ffffffffa070de2b&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] &lt;4&gt;[2551157.909336] [&lt;ffffffffa0d11106&gt;] mdt_enqueue+0x46/0xe0 [mdt] &lt;4&gt;[2551157.916083] [&lt;ffffffffa0d15ada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] &lt;4&gt;[2551157.923701] [&lt;ffffffffa0d52595&gt;] mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt;[2551157.931144] [&lt;ffffffffa073cf25&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] &lt;4&gt;[2551157.940128] [&lt;ffffffffa040a4ce&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] &lt;4&gt;[2551157.947452] [&lt;ffffffffa041b7c5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] &lt;4&gt;[2551157.955380] [&lt;ffffffffa07358f9&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] &lt;4&gt;[2551157.963287] [&lt;ffffffff810546b9&gt;] ? __wake_up_common+0x59/0x90 &lt;4&gt;[2551157.970142] [&lt;ffffffffa073f6ed&gt;] ptlrpc_main+0xaed/0x1930 [ptlrpc] &lt;4&gt;[2551157.977487] [&lt;ffffffffa073ec00&gt;] ? ptlrpc_main+0x0/0x1930 [ptlrpc] &lt;4&gt;[2551157.984809] [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 &lt;4&gt;[2551157.990580] [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 &lt;4&gt;[2551157.998367] [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[2551158.004229] [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[2551158.010272] &lt;0&gt;[2551158.012746] Kernel panic - not syncing: LBUG </pre> </div></div> 2.5.3-2.6.32_431.29.2.el6.atlas.x86_64_g57d5785 LU-6996

osd_ea_lookup_rec assertion

Bug Major Resolved Fixed Alex Zhuravlev Matt Ezell Wed, 12 Aug 2015 20:39:03 +0000 Sat, 16 Jan 2016 06:04:00 +0000 Tue, 6 Oct 2015 14:22:50 +0000 Lustre 2.5.3 Lustre 2.8.0 0 11 <p>We did get a crash dump. The code surrounding the assertion is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lustre/osd-ldiskfs/osd_handler.c</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> osd_ea_lookup_rec(<span class="code-keyword">const</span> struct lu_env *env, struct osd_object *obj, struct dt_rec *rec, <span class="code-keyword">const</span> struct dt_key *key) { struct inode *dir = obj-&gt;oo_inode; ... LASSERT(dir-&gt;i_op != NULL &amp;&amp; dir-&gt;i_op-&gt;lookup != NULL); </pre> </div></div> <p>I loaded this in crash to try to see if it's i_op or lookup that's NULL.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; xbt|grep obj xbt: cannot find debuginfo for module 'libcfs' obj = ffff8815aca40b40 pobj = ffff881bd96f23c0 object_locked = 0 crash&gt; mod -s osd_ldiskfs MODULE NAME SIZE OBJECT FILE ffffffffa0ceaee0 osd_ldiskfs 454984 /lib/modules/2.6.32-431.29.2.el6.atlas.x86_64/extra/kernel/fs/lustre/osd_ldiskfs.ko crash&gt; osd_object.oo_inode ffff8815aca40b40 oo_inode = 0xffff882b54a72a80 crash&gt; inode.i_op 0xffff882b54a72a80 i_op = 0xffffffffa0bf0c40 crash&gt; inode_operations.lookup 0xffffffffa0bf0c40 lookup = 0xffffffffa0bd31f0 crash&gt; dis 0xffffffffa0bd31f0 0xffffffffa0bd31f0 &lt;ldiskfs_lookup&gt;: push %rbp </pre> </div></div> <p>Both look non-NULL and valid.</p> <p>Hi Matt,</p> <p>There might be a race. Could you please dump the stack traces for all tasks? Thank you.</p> <p>Alex</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>The output of 'bt -a' and 'ps' from crash. Let me know if you need the backtrace from any idle PIDs.</p> <p>Thank you, Matt. Alex would look into the stack traces and ask you for help to get more logs if needed.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16026</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6996" title="osd_ea_lookup_rec assertion" class="issue-link" data-issue-key="LU-6996"><del>LU-6996</del></a> osd-ldiskfs: handle stale OI mapping cache<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 4bb351a4e4c76c8538532dcbfb7829dfea35aed0</p> <p>On server side, the RPC service thread may cache one OI mapping on its stack, such OI mapping will become invalid if some other (RPC service thread) removed the object by race. If the RPC service thread uses the cached OI mapping and finds the inode that has been unlinked and reused by other object with no LMA generated yet, then the original osd_check_lma() would regard it as the expect local object by wrong. Such case is one of the reason for the failure in this ticket.</p> <p>Anyway, it is just possible reason, with the given stack/logs, we cannot say that the patch will fix the failure completely. Please apply the patch and see what will happen.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16137" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16137</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6996" title="osd_ea_lookup_rec assertion" class="issue-link" data-issue-key="LU-6996"><del>LU-6996</del></a> osd: test b2_5 base<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: f78965727e7a86786cdb9fe7a15389721be1b1bd</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16157" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16157</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6996" title="osd_ea_lookup_rec assertion" class="issue-link" data-issue-key="LU-6996"><del>LU-6996</del></a> osd-ldiskfs: handle stale OI mapping cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ea2bb60f6b85e53ea43ba240ef1c7e3ef809595c</p> <p>Hi Alex,</p> <p>Nasf is working on the patches handling stale OI mapping cache but he was unsure of the root cause of the original issue in this ticket. Could you please give some more suggestions here?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16157/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16157/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6996" title="osd_ea_lookup_rec assertion" class="issue-link" data-issue-key="LU-6996"><del>LU-6996</del></a> osd-ldiskfs: handle stale OI mapping cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7aaa680b7f22e7dfaac8af38b78d89164a94e842</p> <p>Landed for 2.8</p> Related Development Rank 1|hzxkd3: Rank (Obsolete) 9223372036854775807 Severity