https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6222 Lustre <p>We seem to be having an issue similar to the one described in <a href="https://jira.whamcloud.com/browse/LU-5883" title="DNE II testing: LustreError: 6618:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed" class="issue-link" data-issue-key="LU-5883"><del>LU-5883</del></a>. We're using 2.6.93 on the clients and 2.6.92 on the servers. The problem can be reproduced reliably on our end, so I'd be happy to provide additional logs/diagnostic information (Is there a standard procedure for this? I couldn't seem to find anything via google.) I've included the kernel log messages that were dumped to the console as well as the stack trace below.</p> <p>kernel:LustreError: 13007:0:(statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) ) failed:<br/> kernel:LustreError: 13007:0:(statahead.c:262:sa_kill()) LBUG<br/> PID: 13007 TASK: ffff880b0aea2040 CPU: 25 COMMAND: "rsync"<br/> #0 <span class="error">&#91;ffff881c20ef16f0&#93;</span> machine_kexec at ffffffff8103b5bb<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/arch/x86/kernel/machine_kexec_64.c: 336<br/> #1 <span class="error">&#91;ffff881c20ef1750&#93;</span> crash_kexec at ffffffff810c9852<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/kernel/kexec.c: 1106<br/> #2 <span class="error">&#91;ffff881c20ef1820&#93;</span> panic at ffffffff8152927e<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/kernel/panic.c: 111<br/> #3 <span class="error">&#91;ffff881c20ef18a0&#93;</span> lbug_with_loc at ffffffffa03c5eeb <span class="error">&#91;libcfs&#93;</span><br/> /usr/src/debug/lustre-2.6.93/libcfs/libcfs/linux/linux-debug.c: 175<br/> #4 <span class="error">&#91;ffff881c20ef18c0&#93;</span> revalidate_statahead_dentry at ffffffffa0a0e31d <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/statahead.c: 263<br/> #5 <span class="error">&#91;ffff881c20ef1a00&#93;</span> ll_statahead at ffffffffa0a0e642 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/libcfs/include/libcfs/libcfs_debug.h: 219<br/> #6 <span class="error">&#91;ffff881c20ef1ae0&#93;</span> ll_lookup_it at ffffffffa09f8907 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/namei.c: 541<br/> #7 <span class="error">&#91;ffff881c20ef1ba0&#93;</span> ll_lookup_nd at ffffffffa09f9029 <span class="error">&#91;lustre&#93;</span><br/> /usr/src/debug/lustre-2.6.93/lustre/llite/namei.c: 771<br/> #8 <span class="error">&#91;ffff881c20ef1bf0&#93;</span> do_lookup at ffffffff8119dc65<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1063<br/> #9 <span class="error">&#91;ffff881c20ef1c50&#93;</span> __link_path_walk at ffffffff8119e8f4<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1239<br/> #10 <span class="error">&#91;ffff881c20ef1d30&#93;</span> path_walk at ffffffff8119f40a<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 558<br/> #11 <span class="error">&#91;ffff881c20ef1d70&#93;</span> filename_lookup at ffffffff8119f61b<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1375<br/> #12 <span class="error">&#91;ffff881c20ef1db0&#93;</span> user_path_at at ffffffff811a0747<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/namei.c: 1597<br/> #13 <span class="error">&#91;ffff881c20ef1e80&#93;</span> vfs_fstatat at ffffffff81193bc0<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 84<br/> #14 <span class="error">&#91;ffff881c20ef1ee0&#93;</span> vfs_lstat at ffffffff81193c7e<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 107<br/> #15 <span class="error">&#91;ffff881c20ef1ef0&#93;</span> sys_newlstat at ffffffff81193ca4<br/> /usr/src/debug/kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/fs/stat.c: 257<br/> #16 <span class="error">&#91;ffff881c20ef1f80&#93;</span> system_call_fastpath at ffffffff8100b072<br/> /usr/src/debug////////kernel-2.6.32-504.3.3.el6/linux-2.6.32-504.3.3.el6.x86_64/arch/x86/kernel/entry_64.S: 489<br/> RIP: 00000031f78daf75 RSP: 00007fff51e03da8 RFLAGS: 00010206<br/> RAX: 0000000000000006 RBX: ffffffff8100b072 RCX: 00007fff51e040b0<br/> RDX: 00007fff51dffb10 RSI: 00007fff51dffb10 RDI: 00007fff51e00ba0<br/> RBP: 0000000000000000 R8: 00007fff51e00be3 R9: 0000000000000000<br/> R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff51e00ba0<br/> R13: 00007fff51dffb10 R14: 00007fff51dffb10 R15: 00007fff51e00ba0<br/> ORIG_RAX: 0000000000000006 CS: 0033 SS: 002b</p> 2.6.93 on the clients and 2.6.92 on the servers, centos with rpms from lustre-master jenkins tree. LU-6222

LustreError (statahead.c:262:sa_kill()) ASSERTION( !list_empty(&entry->se_list) )

Bug Minor Resolved Fixed Lai Siyao Andrew Zenk Fri, 6 Feb 2015 17:36:23 +0000 Mon, 8 Jun 2015 07:50:48 +0000 Fri, 13 Feb 2015 05:20:27 +0000 Lustre 2.7.0 0 7 <p>Lai,<br/> Could you please have a look at this one and comment?<br/> Thank you!</p> <p>Can you please detail your reproduction steps if this is something we can easily replicate?<br/> Do you use DNE? What's your exact configuration of Lustre?</p> <p>We do not use DNE. In this case a user is running an rsync as follows "rsync --size-only --progress -av --prune-empty-dirs --include=/ --exclude=.man --exclude=<b>_eot.txt --exclude=.MAN --exclude=_EOT.TXT --include=052903541090_01</b>* --exclude=* /lustre_mountpoint/staging/orig/_uploads/DG_A11281 /lustre_mountpoint/somepath/northslope/" This command is run several times in a serial fashion from a single client with slight variations of the include and srcdir for each run. At a some, seemingly random point during the sequence of rsync jobs, the kernel on the client node panics. </p> <p>Configuration:<br/> We're using a single MDS with nine OSSs. Each OSS contains 3 direct attached targets. Our system isn't designed for failover. The MDS and OSS nodes are booted from a common image built using centos 6.6 with this build: <a href="https://build.hpdd.intel.com/job/lustre-master/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/2832/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-master/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/2832/artifact/artifacts/RPMS/x86_64/</a>. </p> <p>I'm happy to supply exact specs on raid configurations and disk counts if you feel that it's important, but we'll skip that for now. The MDS is using a single target on a ssd raid10. The OSTs are sata of various types. All servers are connected to our QDR IB fabric as well as a gigabit VLAN. The latter is used for connecting 3 clients that aren't experiencing any issues. </p> <p>There are approximately 20 clients, which are also using centos 6.6. The lustre clients are installed via the pre-built rpms from lustre-master, just like the servers. Though the clients are of slightly mixed build versions. The two clients that we've reproduced the issue on were both running build #2835.</p> <p>The entire filesystem has a stripe count of 1.</p> <p>Let me know if you need any additional information. Thanks!</p> <p>could you list all process backtraces in the dump?</p> <p>Attached output from foreach bt -l </p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13708" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13708</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6222" title="LustreError (statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) )" class="issue-link" data-issue-key="LU-6222"><del>LU-6222</del></a> statahead: add to list before make ready<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c77d1c16d45850e3cd4492d27746b30a89ba2beb</p> <p>Andrew, I just uploaded a fix for this issue, will you apply it and test again?</p> <p>Thanks! We're testing it now.</p> <p>That seems to have fixed it. The rsync script that consistently caused the issue after a minute or two has been running flawlessly for many hours. Thanks again.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13708/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13708/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6222" title="LustreError (statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) )" class="issue-link" data-issue-key="LU-6222"><del>LU-6222</del></a> statahead: add to list before make ready<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c59baf3ff10beca9841bc8aae211af120ab913dc</p> <p>Landed for 2.7</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15178" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15178</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6222" title="LustreError (statahead.c:262:sa_kill()) ASSERTION( !list_empty(&amp;entry-&gt;se_list) )" class="issue-link" data-issue-key="LU-6222"><del>LU-6222</del></a> statahead: add to list before make ready<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 1091da447db103549ca8e20d5c5fb97679f7080c</p> Development Rank 1|hzx5uf: Rank (Obsolete) 17399 Severity