https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5883 Lustre <p>While doing a general purpose test of master with DNE II (2 MDSes with 3 MDTs each, 6 total MDTs), we did an ls to check the status of something, and our client LBUGged. We turned up debug on a different client, did the ls again, and it crashed as well:</p> <p>2014-11-06T20:08:43.974554-06:00 c1-0c0s0n3 LustreError: 3797:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed:<br/> 2014-11-06T20:08:43.974591-06:00 c1-0c0s0n3 LustreError: 3797:0:(statahead.c:262:sa_kill()) LBUG<br/> 2014-11-06T20:08:43.974597-06:00 c1-0c0s0n3 Pid: 3797, comm: ls<br/> 2014-11-06T20:08:43.974604-06:00 c1-0c0s0n3 Call Trace:<br/> 2014-11-06T20:08:43.974617-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81006591&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2014-11-06T20:08:43.974622-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81004de9&gt;&#93;</span> dump_trace+0x89/0x440<br/> 2014-11-06T20:08:43.974632-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa0176897&gt;&#93;</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2014-11-06T20:08:43.974639-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa0176de7&gt;&#93;</span> lbug_with_loc+0x47/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2014-11-06T20:08:44.015207-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa0750512&gt;&#93;</span> sa_put+0x332/0x370 <span class="error">&#91;lustre&#93;</span><br/> 2014-11-06T20:08:44.015232-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa0752b6c&gt;&#93;</span> do_statahead_enter+0xfdc/0x1cb0 <span class="error">&#91;lustre&#93;</span><br/> 2014-11-06T20:08:44.015238-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa073a695&gt;&#93;</span> ll_lookup_it+0x7b5/0x1b70 <span class="error">&#91;lustre&#93;</span><br/> 2014-11-06T20:08:44.015245-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffffa073bad5&gt;&#93;</span> ll_lookup_nd+0x85/0x570 <span class="error">&#91;lustre&#93;</span><br/> 2014-11-06T20:08:44.015250-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81148edc&gt;&#93;</span> d_alloc_and_lookup+0x4c/0x80<br/> 2014-11-06T20:08:44.015261-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8114a57e&gt;&#93;</span> do_lookup+0x29e/0x3a0<br/> 2014-11-06T20:08:44.015273-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8114cdb3&gt;&#93;</span> path_lookupat+0xc3/0x5e0<br/> 2014-11-06T20:08:44.015281-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8114d305&gt;&#93;</span> do_path_lookup+0x35/0xd0<br/> 2014-11-06T20:08:44.015287-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8114e043&gt;&#93;</span> user_path_at_empty+0x83/0xb0<br/> 2014-11-06T20:08:44.015303-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8114e081&gt;&#93;</span> user_path_at+0x11/0x20<br/> 2014-11-06T20:08:44.015311-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81142d85&gt;&#93;</span> vfs_fstatat+0x55/0x90<br/> 2014-11-06T20:08:44.054257-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81142e2e&gt;&#93;</span> vfs_lstat+0x1e/0x20<br/> 2014-11-06T20:08:44.054275-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff81142e54&gt;&#93;</span> sys_newlstat+0x24/0x50<br/> 2014-11-06T20:08:44.054281-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;ffffffff8133776b&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> 2014-11-06T20:08:44.054293-06:00 c1-0c0s0n3 <span class="error">&#91;&lt;00007ffff78b9515&gt;&#93;</span> 0x7ffff78b9515</p> <p>I'll make the client dump available in a few minutes.</p> Versions of 2.6.54 on clients &amp; servers.<br/> Cray SLES11SP3 clients, CentOS servers (2.6.32-431.5.1.el6.x86_64).<br/> <br/> Most recent commit on clients:<br/> Ie7a2a98be8cc97db9af7a64476c06fc7321544eb<br/> <a href="http://review.whamcloud.com/12142">http://review.whamcloud.com/12142</a><br/> <br/> Most recent commit on servers:<br/> If24443955290b091fd22905dfb74b0d6a6d1b4e8<br/> <a href="http://review.whamcloud.com/12490">http://review.whamcloud.com/12490</a> LU-5883

DNE II testing: LustreError: 6618:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&entry->se_list) ) failed

Bug Major Resolved Cannot Reproduce Lai Siyao Patrick Farrell Fri, 7 Nov 2014 15:57:06 +0000 Sat, 9 Oct 2021 06:42:11 +0000 Sat, 9 Oct 2021 06:42:11 +0000 Lustre 2.7.0 0 5 <p>Dump and associated kos (and system logs for that node) going up here:<br/> ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5883" title="DNE II testing: LustreError: 6618:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed" class="issue-link" data-issue-key="LU-5883"><del>LU-5883</del></a>/c1-0c0s1n2-20141107.tar.gz</p> <p>The node had full debugging enabled when it was crashed.</p> <p>Confirmed from the engineer testing it - The ls was of the root of the file system. There should not have been any remote directories in the root of the file system, so this may have nothing to do with DNE II.</p> <p>Full Lustre log extracted from the client is here:<br/> ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5883" title="DNE II testing: LustreError: 6618:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed" class="issue-link" data-issue-key="LU-5883"><del>LU-5883</del></a>/c1-0c0s1n2.lustre.log</p> <p>Note that, weirdly, dirdata was off on MDT0. This system was used to troubleshoot <a href="https://jira.whamcloud.com/browse/LU-5626" title="Corruption of MDT “..” entry in non-HTree ldiskfs directories" class="issue-link" data-issue-key="LU-5626"><del>LU-5626</del></a> and we forgot to reenable dirdata. We'll reenable it before the next run.</p> <p>From the debug log it seems not related with DNE, hmm, I saw </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-3270 statahead: statahead thread wait for RPCs to finish Statahead thread should wait for inflight stat RPCs to finish in case statahead RPC callback may access data allocated in statahead thread context. ll_sa_entry_fini() should keep old entry if stat RPC is not finished yet. Simplify sai refcounting: * newly allocated sai will hold one refcount, and it will put it after starting statahead thread. * statahead thread holds one refcount. * agl thread holds one refcount. * stat process calls do_statahead_enter() which will try to get sai, and if it's valid, it will revalidate from statahead cache, and put refcount after use. Signed-off-by: Lai Siyao &lt;lai.siyao@intel.com&gt; Change-Id: I55a4fe66a5f6c04595d3bc84f0cd3750f20e0ee4 Reviewed-on: http://review.whamcloud.com/9663 Tested-by: Jenkins Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt; Reviewed-by: Fan Yong &lt;fan.yong@intel.com&gt; Reviewed-by: James Simmons &lt;uja.ornl@gmail.com&gt; Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt; </pre> </div></div> <p>Just changed these area in 2.6. Lai, could you please have a look. Thanks.</p> <p>this is a known issue: revalidate_statahead_dentry() may fail to wait for a statahead entry to become ready, in this case it should not release this entry. originally this was found in 2.4 statahead patches, and I'm waiting for their response. I'll update the last patch of <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a>: <a href="http://review.whamcloud.com/#/c/9667/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9667/</a>, so if you want to test statahead, it's better to apply this one and test.</p> <p>Patch updated, if test is okay, I'll mark this duplicate.</p> <p>Thanks, Lai - We'll use that patch when we next update the system. That will be later this week, I'll try to give test results shortly after that.</p> <p>I see that the patch seems to have been applied, though we seem to be having a similar problem. We're using 2.6.93 on the clients and 2.6.92 on the servers. The problem can be reproduced reliably on our end, so I'd be happy to provide additional logs/diagnostic information (Is there a standard procedure for this? I couldn't seem to find anything via google.) I've included the kernel log messages that were dumped to the console as well as the stack trace below.</p> <p> kernel:LustreError: 13007:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed:<br/> kernel:LustreError: 13007:0:(statahead.c:262:sa_kill()) LBUG</p> <p>PID: 13007 TASK: ffff880b0aea2040 CPU: 25 COMMAND: "rsync"<br/> #0 <span class="error">&#91;ffff881c20ef16f0&#93;</span> machine_kexec at ffffffff8103b5bb<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/arch/x86/kernel/machine_kexec_64.c: 336<br/> #1 <span class="error">&#91;ffff881c20ef1750&#93;</span> crash_kexec at ffffffff810c9852<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/kernel/kexec.c: 1106<br/> #2 <span class="error">&#91;ffff881c20ef1820&#93;</span> panic at ffffffff8152927e<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/kernel/panic.c: 111<br/> #3 <span class="error">&#91;ffff881c20ef18a0&#93;</span> lbug_with_loc at ffffffffa03c5eeb <span class="error">&#91;libcfs&#93;</span><br/> /usr/src/debug/lustre-2.6.93/libcfs/libcfs/linux/linux-debug.c: 175<br/> #4 <span class="error">&#91;ffff881c20ef18c0&#93;</span> revalidate_statahead_dentry at ffffffffa0a0e31d <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/statahead.c: 263<br/> #5 <span class="error">&#91;ffff881c20ef1a00&#93;</span> ll_statahead at ffffffffa0a0e642 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/libcfs/include/libcfs/libcfs_debug.h: 219<br/> #6 <span class="error">&#91;ffff881c20ef1ae0&#93;</span> ll_lookup_it at ffffffffa09f8907 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/namei.c: 541<br/> #7 <span class="error">&#91;ffff881c20ef1ba0&#93;</span> ll_lookup_nd at ffffffffa09f9029 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/namei.c: 771<br/> #8 <span class="error">&#91;ffff881c20ef1bf0&#93;</span> do_lookup at ffffffff8119dc65<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1063<br/> #9 <span class="error">&#91;ffff881c20ef1c50&#93;</span> __link_path_walk at ffffffff8119e8f4<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1239<br/> #10 <span class="error">&#91;ffff881c20ef1d30&#93;</span> path_walk at ffffffff8119f40a<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 558<br/> #11 <span class="error">&#91;ffff881c20ef1d70&#93;</span> filename_lookup at ffffffff8119f61b<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1375<br/> #12 <span class="error">&#91;ffff881c20ef1db0&#93;</span> user_path_at at ffffffff811a0747<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1597<br/> #13 <span class="error">&#91;ffff881c20ef1e80&#93;</span> vfs_fstatat at ffffffff81193bc0<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 84<br/> #14 <span class="error">&#91;ffff881c20ef1ee0&#93;</span> vfs_lstat at ffffffff81193c7e<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 107<br/> #15 <span class="error">&#91;ffff881c20ef1ef0&#93;</span> sys_newlstat at ffffffff81193ca4<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 257<br/> #16 <span class="error">&#91;ffff881c20ef1f80&#93;</span> system_call_fastpath at ffffffff8100b072<br/> /usr/src/debug////////kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/arch/x86/kernel/entry_64.S: 489<br/> RIP: 00000031f78daf75 RSP: 00007fff51e03da8 RFLAGS: 00010206<br/> RAX: 0000000000000006 RBX: ffffffff8100b072 RCX: 00007fff51e040b0<br/> RDX: 00007fff51dffb10 RSI: 00007fff51dffb10 RDI: 00007fff51e00ba0<br/> RBP: 0000000000000000 R8: 00007fff51e00be3 R9: 0000000000000000<br/> R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff51e00ba0<br/> R13: 00007fff51dffb10 R14: 00007fff51dffb10 R15: 00007fff51e00ba0<br/> ORIG_RAX: 0000000000000006 CS: 0033 SS: 002b</p> Development Rank 1|hzx0gv: Rank (Obsolete) 16451 Severity