https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4692 Lustre <p>The assertion failed on our system. The system are running Lustre-2.5.54, so patch from <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> is already merged.</p> <p>&lt;3&gt;LustreError: 109372:0:(file.c:3087:ll_inode_revalidate_fini()) home2: revalidate FID <span class="error">&#91;0x2000048a2:0x17:0x0&#93;</span> error: rc = -116<br/> &lt;0&gt;LustreError: 237806:0:(osc_lock.c:1204:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 6<br/> &lt;0&gt;LustreError: 237806:0:(osc_lock.c:1204:osc_lock_enqueue()) LBUG<br/> &lt;4&gt;Pid: 237806, comm: tail<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa054c895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa054ce97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c620f0&gt;&#93;</span> ? osc_lock_enqueue+0x0/0x8c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c62820&gt;&#93;</span> osc_lock_enqueue+0x730/0x8c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069a337&gt;&#93;</span> ? cl_lock_state_signal+0x87/0x160 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069dcfc&gt;&#93;</span> cl_enqueue_try+0xfc/0x300 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0a5c42a&gt;&#93;</span> lov_lock_enqueue+0x22a/0x850 <span class="error">&#91;lov&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069dcfc&gt;&#93;</span> cl_enqueue_try+0xfc/0x300 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069ef4f&gt;&#93;</span> cl_enqueue_locked+0x6f/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069fb9e&gt;&#93;</span> cl_lock_request+0x7e/0x270 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06a4b2c&gt;&#93;</span> cl_io_lock+0x3cc/0x560 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06a4d62&gt;&#93;</span> cl_io_loop+0xa2/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0addca6&gt;&#93;</span> ll_file_io_generic+0x2b6/0x710 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0694ce9&gt;&#93;</span> ? cl_env_get+0x29/0x350 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ade23f&gt;&#93;</span> ll_file_aio_read+0x13f/0x2c0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ade6fc&gt;&#93;</span> ll_file_read+0x16c/0x2a0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff811816c5&gt;&#93;</span> vfs_read+0xb5/0x1a0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81181801&gt;&#93;</span> sys_read+0x51/0x90<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> &lt;4&gt;<br/> &lt;0&gt;Kernel panic - not syncing: LBUG<br/> &lt;4&gt;Pid: 237806, comm: tail Tainted: G W --------------- 2.6.32-358.el6.x86_64 #1<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8150cfc8&gt;&#93;</span> ? panic+0xa7/0x16f<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa054ceeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c620f0&gt;&#93;</span> ? osc_lock_enqueue+0x0/0x8c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c62820&gt;&#93;</span> ? osc_lock_enqueue+0x730/0x8c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069a337&gt;&#93;</span> ? cl_lock_state_signal+0x87/0x160 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069dcfc&gt;&#93;</span> ? cl_enqueue_try+0xfc/0x300 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0a5c42a&gt;&#93;</span> ? lov_lock_enqueue+0x22a/0x850 <span class="error">&#91;lov&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069dcfc&gt;&#93;</span> ? cl_enqueue_try+0xfc/0x300 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069ef4f&gt;&#93;</span> ? cl_enqueue_locked+0x6f/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa069fb9e&gt;&#93;</span> ? cl_lock_request+0x7e/0x270 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06a4b2c&gt;&#93;</span> ? cl_io_lock+0x3cc/0x560 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06a4d62&gt;&#93;</span> ? cl_io_loop+0xa2/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0addca6&gt;&#93;</span> ? ll_file_io_generic+0x2b6/0x710 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0694ce9&gt;&#93;</span> ? cl_env_get+0x29/0x350 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ade23f&gt;&#93;</span> ? ll_file_aio_read+0x13f/0x2c0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ade6fc&gt;&#93;</span> ? ll_file_read+0x16c/0x2a0 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff811816c5&gt;&#93;</span> ? vfs_read+0xb5/0x1a0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81181801&gt;&#93;</span> ? sys_read+0x51/0x90<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> 2.5.54 LU-4692

(osc_lock.c:1204:osc_lock_enqueue()) ASSERTION( ols->ols_state == OLS_NEW ) failed: Impossible state: 6

Bug Critical Resolved Duplicate Zhenyu Xu Li Xi Mon, 3 Mar 2014 07:09:36 +0000 Wed, 12 Nov 2014 18:08:38 +0000 Thu, 11 Sep 2014 00:53:26 +0000 Lustre 2.4.3 0 14 <p>Bobijam</p> <p>Could you pleases assess this issue reported against master?</p> <p>Thanks</p> <p>Peter</p> <p>when an IO is starting, it will check whether there are existing locks matching to what this IO needs, the calling path is</p> <p>cl_lockset_lock_one()=&gt;cl_lock_request()=&gt;cl_lock_hold_mutex()=&gt;cl_lock_find()=&gt;cl_lock_lookup()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> matched = cl_lock_ext_match(&amp;lock-&gt;cll_descr, need) &amp;&amp; lock-&gt;cll_state &lt; CLS_FREEING &amp;&amp; lock-&gt;cll_error == 0 &amp;&amp; !(lock-&gt;cll_flags &amp; CLF_CANCELLED) &amp;&amp; cl_lock_fits_into(env, lock, need, io); </pre> </div></div> <p>and in osc_lock_fits_into()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct osc_lock *ols = cl2osc_lock(slice); <span class="code-keyword">if</span> (need-&gt;cld_enq_flags &amp; CEF_NEVER) <span class="code-keyword">return</span> 0; <span class="code-keyword">if</span> (ols-&gt;ols_state &gt;= OLS_CANCELLED) <span class="code-keyword">return</span> 0; </pre> </div></div> <p>So means after osc_lock_fits_into() checking, the osc_lock was cancelled by another thread, so that the new IO think it finds a fitting lock, and when it tries to enqueue the lock, it finds that osc_lock is in OLS_CANCELLED instead of OLS_NEW.</p> <p>The lock will be held in this case so it won't be cancelled. </p> <p>Let's make up a scenario, thread1 is starting an IO, thread2 has the fitting lock (both top_lock and sub_lock), and thread2 is handling osc lock cancel ast requested from server</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>thread1 thread2 note cl_lock_request() +-&gt;cl_lock_hold_mutex() +--&gt;cl_lock_find() found the fitting top lock, and its sub osc lock is not in OLS_CANCELLED | | osc_ldlm_blocking_ast() | +-&gt;osc_ldlm_blocking_ast0() get mutex of sub lock | |+-&gt;osc_lock_blocking() | | +-&gt;cl_lock_cancel(sub_lock) | | +-&gt;cl_lock_cancel0(sub_lock) set sub_lock as CLF_CANCELLED, reverse call clo_cancel() | | |+-&gt;osc_lock_cancel(sub_osc_lock) set osc_lock as OLS_CANCELLED | | +-&gt;cl_lock_delete(sub_lock) | +-&gt;cl_lock_mutex_put(sub_lock) | | cl_lock_mutex_get(top_lock) </pre> </div></div> <p>at this time, top_lock is not in CLF_CANCELLED while sub osc_lock is in OLS_CANCELLED state.</p> <p>Li Xi, could you please comment about what test was being run to cause this problem?</p> <p>This is probably a different representation of <a href="https://jira.whamcloud.com/browse/LU-4591" title="Related cl_lock failures on master/2.5" class="issue-link" data-issue-key="LU-4591"><del>LU-4591</del></a>. It would be really helpful if someone on our team can help reproduce the problem local side.</p> <p>Hi Andreas,</p> <p>Unfortunately, I don't know what test was running. I will let you know as soon as I get more information about ot.</p> <p>We hit the same problem shortly after upgrading to 2.4.2 clients. Have not seen this on 2.4.1</p> <p>crash&gt; bt<br/> PID: 18546 TASK: ffff8809f1aa8040 CPU: 0 COMMAND: "turbostream"<br/> #0 <span class="error">&#91;ffff8809f0cf98a0&#93;</span> machine_kexec at ffffffff81038f3b<br/> #1 <span class="error">&#91;ffff8809f0cf9900&#93;</span> crash_kexec at ffffffff810c5da2<br/> #2 <span class="error">&#91;ffff8809f0cf99d0&#93;</span> panic at ffffffff8152721a<br/> #3 <span class="error">&#91;ffff8809f0cf9a50&#93;</span> lbug_with_loc at ffffffffa0f47eeb <span class="error">&#91;libcfs&#93;</span><br/> #4 <span class="error">&#91;ffff8809f0cf9a70&#93;</span> lovsub_lock_state at ffffffffa14b7d51 <span class="error">&#91;lov&#93;</span><br/> #5 <span class="error">&#91;ffff8809f0cf9ac0&#93;</span> cl_lock_state_signal at ffffffffa10aedd8 <span class="error">&#91;obdclass&#93;</span><br/> #6 <span class="error">&#91;ffff8809f0cf9b00&#93;</span> cl_lock_state_set at ffffffffa10aef25 <span class="error">&#91;obdclass&#93;</span><br/> #7 <span class="error">&#91;ffff8809f0cf9b30&#93;</span> cl_enqueue_try at ffffffffa10b2c29 <span class="error">&#91;obdclass&#93;</span><br/> #8 <span class="error">&#91;ffff8809f0cf9b80&#93;</span> lov_lock_enqueue at ffffffffa14b00da <span class="error">&#91;lov&#93;</span><br/> #9 <span class="error">&#91;ffff8809f0cf9c20&#93;</span> cl_enqueue_try at ffffffffa10b2bdc <span class="error">&#91;obdclass&#93;</span><br/> #10 <span class="error">&#91;ffff8809f0cf9c70&#93;</span> cl_enqueue_locked at ffffffffa10b3fcf <span class="error">&#91;obdclass&#93;</span><br/> #11 <span class="error">&#91;ffff8809f0cf9cb0&#93;</span> cl_lock_request at ffffffffa10b4c3e <span class="error">&#91;obdclass&#93;</span><br/> #12 <span class="error">&#91;ffff8809f0cf9d10&#93;</span> cl_io_lock at ffffffffa10b9edc <span class="error">&#91;obdclass&#93;</span><br/> #13 <span class="error">&#91;ffff8809f0cf9d70&#93;</span> cl_io_loop at ffffffffa10ba112 <span class="error">&#91;obdclass&#93;</span><br/> #14 <span class="error">&#91;ffff8809f0cf9da0&#93;</span> ll_file_io_generic at ffffffffa152ca90 <span class="error">&#91;lustre&#93;</span><br/> #15 <span class="error">&#91;ffff8809f0cf9e20&#93;</span> ll_file_aio_write at ffffffffa152d9d2 <span class="error">&#91;lustre&#93;</span><br/> #16 <span class="error">&#91;ffff8809f0cf9e80&#93;</span> ll_file_write at ffffffffa152dcbc <span class="error">&#91;lustre&#93;</span><br/> #17 <span class="error">&#91;ffff8809f0cf9ef0&#93;</span> vfs_write at ffffffff81188f88<br/> #18 <span class="error">&#91;ffff8809f0cf9f30&#93;</span> sys_write at ffffffff81189881<br/> #19 <span class="error">&#91;ffff8809f0cf9f80&#93;</span> system_call_fastpath at ffffffff8100b072<br/> RIP: 00007fdf7eeea6fd RSP: 00007fffad51c8b8 RFLAGS: 00000202<br/> RAX: 0000000000000001 RBX: ffffffff8100b072 RCX: 0000000000000001<br/> RDX: 00000000000009c4 RSI: 0000000008b32154 RDI: 000000000000001b<br/> RBP: 0000000013252a00 R8: 00007fdf8069d6a8 R9: 00000000000009c4<br/> R10: 00007fffad512d3c R11: 0000000000000293 R12: 000000004c00010d<br/> R13: 00000000000009c4 R14: 0000000000e6d964 R15: 000000000000001b<br/> ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b<br/> crash&gt; sys<br/> SYSTEM MAP: /boot/System.map-2.6.32-431.1.2.el6.hpcs.x86_64<br/> DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.1.2.el6.hpcs.x86_64/vmlinux (2.6.32-431.1.2.el6.hpcs.x86_64)<br/> DUMPFILE: vmcore <span class="error">&#91;PARTIAL DUMP&#93;</span><br/> CPUS: 12<br/> DATE: Wed Mar 12 16:17:17 2014<br/> UPTIME: 1 days, 23:37:03<br/> LOAD AVERAGE: 1.99, 1.97, 1.91<br/> TASKS: 449<br/> NODENAME: tesla14<br/> RELEASE: 2.6.32-431.1.2.el6.hpcs.x86_64<br/> VERSION: #1 SMP Sun Dec 15 04:51:47 GMT 2013<br/> MACHINE: x86_64 (2599 Mhz)<br/> MEMORY: 64 GB<br/> PANIC: "Kernel panic - not syncing: LBUG"</p> <p>Hi Turek,</p> <p>Can you share us the core dump? You can upload it to ftp.whamcloud.com, thanks.</p> <p>Jinshan</p> <p>uploaded to <a href="ftp://ftp.whamcloud.com/uploads/LU-4692/" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/LU-4692/</a></p> <p>Hi Wojciech,</p> <p>Can you upload the supporting files (uncompressed vmlinuz, System.map) as well?</p> <p>Here is crashdump of original this problem. <a href="ftp://ftp.whamcloud.com/uploads/LU-4692-2/" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/LU-4692-2/</a><br/> The kernel is standard RHEL6.4, you can download vmlinux and System.map from here.<br/> <a href="http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-358.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-358.el6.x86_64.rpm</a><br/> <a href="http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-358.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-x86_64-2.6.32-358.el6.x86_64.rpm</a></p> <p>we use patchless clients so the kernel is standard rhel6.5 kernel<br/> <a href="ftp://ftp.scientificlinux.org/linux/scientific/6.5/archive/debuginfo/kernel-debuginfo-2.6.32-431.1.2.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.scientificlinux.org/linux/scientific/6.5/archive/debuginfo/kernel-debuginfo-2.6.32-431.1.2.el6.x86_64.rpm</a><br/> <a href="ftp://ftp.scientificlinux.org/linux/scientific/6.5/archive/debuginfo/kernel-debuginfo-common-x86_64-2.6.32-431.1.2.el6.x86_64.rpm" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.scientificlinux.org/linux/scientific/6.5/archive/debuginfo/kernel-debuginfo-common-x86_64-2.6.32-431.1.2.el6.x86_64.rpm</a></p> <p>This problem should be imported by <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a>, I'm taking a look.</p> <p>I think this is imported by <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>, I pushed <a href="http://review.whamcloud.com/9783" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9783</a> please give it a try.</p> <p>It lacks of lustre module files so I can't analyze the core dump file.</p> <p>Jinshan, <br/> Wojciech seems to be running 2.4.2, as we do. And we both hit this bug and I'm sure we did not apply <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> on top of it. So it is possible to hit this bug without <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>. So I do not think it is responsible for this issue.</p> <p>I see, thanks Aurelien.</p> <p>Hi Wojciech, please upload lustre modules into ftp so I can take a further look at the core dump.</p> <p>The first part of this bug was related to <a href="https://jira.whamcloud.com/browse/LU-4591" title="Related cl_lock failures on master/2.5" class="issue-link" data-issue-key="LU-4591"><del>LU-4591</del></a></p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-5492" title="osc_lock.c:1165:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 6" class="issue-link" data-issue-key="LU-5492"><del>LU-5492</del></a></p> Duplicate LU-4443 Related LU-5492 LU-4591 Development Rank 1|hzwggn: Rank (Obsolete) 12901 Severity