https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3433 Lustre <p>During a test run using Lustre 2.4 one of our clients encountered this LBUG.</p> <p><span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span>LustreError:16573:0:(osc_lock.c:1165:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 4<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span>LustreError: 16573:0<br/> (osc_lock.c:1165:osc_lock_enqueue()) LBUG<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff81006451&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff81004ca9&gt;&#93;</span> dump_trace+0x89/0x440<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa013a897&gt;&#93;</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa013ade7&gt;&#93;</span> lbug_with_loc+0x47/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa0639f55&gt;&#93;</span> osc_lock_enqueue+0x725/0x8b0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa032d4eb&gt;&#93;</span> cl_enqueue_try+0xfb/0x320 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa06ccded&gt;&#93;</span> lov_lock_enqueue+0x1fd/0x880 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa032d4eb&gt;&#93;</span> cl_enqueue_try+0xfb/0x320 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa032e3bf&gt;&#93;</span> cl_enqueue_locked+0x7f/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa032efbe&gt;&#93;</span> cl_lock_request+0x7e/0x270 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa0334274&gt;&#93;</span> cl_io_lock+0x394/0x5c0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa033453a&gt;&#93;</span> cl_io_loop+0x9a/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa074b90f&gt;&#93;</span> ll_file_io_generic+0x33f/0x5f0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa074c08b&gt;&#93;</span> ll_file_aio_read+0x23b/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffffa074d002&gt;&#93;</span> ll_file_read+0x1f2/0x280 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff81135548&gt;&#93;</span> vfs_read+0xc8/0x180<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff8113b799&gt;&#93;</span> kernel_read+0x49/0x60<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff8113b885&gt;&#93;</span> prepare_binprm+0xd5/0x100<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff8113c670&gt;&#93;</span> do_execve_common+0x1c0/0x300<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff8113c83f&gt;&#93;</span> do_execve+0x3f/0x50<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff8100ae1e&gt;&#93;</span> sys_execve+0x4e/0x80<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;ffffffff81316a3c&gt;&#93;</span> stub_execve+0x6c/0xc0<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span> <span class="error">&#91;&lt;0000000020176437&gt;&#93;</span> 0x20176437<br/> <span class="error">&#91;2013-05-29 18:39:47&#93;</span><span class="error">&#91;c7-1c1s3n3&#93;</span>Kernel panic - not syncing: LBUG</p> Cray Lustre 2.4 clients running on SLES11 SP2. LU-3433

Encountered a assertion for the ols_state being set to a impossible state

Bug Major Resolved Fixed Niu Yawei James A Simmons mn4 Mon, 3 Jun 2013 13:56:17 +0000 Wed, 12 Nov 2014 18:07:18 +0000 Thu, 20 Feb 2014 15:36:24 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 0 16 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Cray has recently noticed this LBUG while testing for a possible different source for the EINTR reported in <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>. A reproducer very similar to that one can consistently produce this bug during writes or truncates. I have not tested reads, but I'm inclined to believe it would reproduce the bug for reads as well.</p> <p>I'll attach the reproducer for this bug for write/truncate.</p> <p>Briefly, the reproducer spawns a child thread, then the parent sets up a signal handler for SIGINT with SA_RESTART set, and loops the IO operation in question (It's set up for write or truncate, and could be modified to use read), while the child spams SIGINT at the parent.</p> <p>This bug is much easier to reproduce with +trace debugging enabled. Without, it usually requires looping the reproducer, sometimes for several minutes. The provided shell script loops the reproducer.</p> <p>This looks like a regression introduced by <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>, when I removed the fix of <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>, the LBUG can't be reproduced anymore, but I don't know the exact reason yet. Patrick, can you reproduce it after removing the fix of <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>?</p> <p>Do you have a reproducer for the <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a> as well. It would be nice to make both conditions work.</p> <p>James,</p> <p>It's the same reproducer, actually. This one - as long as it's in write mode and not truncate <span class="error">&#91;it might work in truncate, I haven&#39;t tested&#93;</span> - should reproduce <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a> just fine. All I changed was to tweak the timings to hit this LBUG more reliably. Without the patch for <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>, <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a> was quite easy to hit.</p> <p>Niu,</p> <p>I'm not sure if I could reproduce it - I can try a bit later. Without <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>, the system call is not restarted, because it returns EINTR. That changes the timing and behavior profile immensely. I don't think this was 'caused' by <a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a>, rather it was revealed by it.</p> <p>Presumably, when a system call is restarted, sometimes we wind up in this nasty state. If EINTR is returned instead (<a href="https://jira.whamcloud.com/browse/LU-3020" title="Lustre returns EINTR during writes when SA_RESTART is set" class="issue-link" data-issue-key="LU-3020"><del>LU-3020</del></a> removed), then the system call is not restarted. It's possible we only get to this state through restarting, or it's possible that we only get there through interrupting the system call at a very specific time, regardless of the restart behavior.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Niu, James,</p> <p>We have a candidate patch for this from Xyratex that we're testing on our end. It appears to resolve the issue, and once I'm sure it's solid, we'll make sure it's pushed upstream.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p><a href="http://review.whamcloud.com/6709" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6709</a></p> <p>patch landed for 2.5</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/66/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/66/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The same failure occurred while running parallel-scale test write_disjoint:<br/> <a href="https://maloo.whamcloud.com/test_sets/e255840a-6193-11e3-a871-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e255840a-6193-11e3-a871-52540035b04c</a></p> <p>Console log on client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>17:35:46:Lustre: DEBUG MARKER: == parallel-scale test write_disjoint: write_disjoint == 17:31:44 (1386639104) 17:35:46:LustreError: 12360:0:(osc_lock.c:1165:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 6 17:35:46:LustreError: 12360:0:(osc_lock.c:1165:osc_lock_enqueue()) LBUG 17:35:46:Pid: 12360, comm: write_disjoint 17:35:47: 17:35:47:Call Trace: 17:35:47: [&lt;ffffffffa19ac895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 17:35:47: [&lt;ffffffffa19ace97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 17:35:47: [&lt;ffffffffa05e4fc0&gt;] ? osc_lock_enqueue+0x0/0x870 [osc] 17:35:47: [&lt;ffffffffa05e5674&gt;] osc_lock_enqueue+0x6b4/0x870 [osc] 17:35:47: [&lt;ffffffffa19ad717&gt;] ? cfs_waitq_broadcast+0x17/0x20 [libcfs] 17:35:47: [&lt;ffffffffa1b15f5e&gt;] ? cl_lock_state_signal+0x7e/0x160 [obdclass] 17:35:47: [&lt;ffffffffa1b19d4c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] 17:35:47: [&lt;ffffffffa06620da&gt;] lov_lock_enqueue+0x22a/0x850 [lov] 17:35:47: [&lt;ffffffffa1b19d4c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] 17:35:47: [&lt;ffffffffa1b1b13f&gt;] cl_enqueue_locked+0x6f/0x1f0 [obdclass] 17:35:47: [&lt;ffffffffa1b1bdae&gt;] cl_lock_request+0x7e/0x270 [obdclass] 17:35:47: [&lt;ffffffffa1b2104c&gt;] cl_io_lock+0x3cc/0x560 [obdclass] 17:35:48: [&lt;ffffffffa1b21282&gt;] cl_io_loop+0xa2/0x1b0 [obdclass] 17:35:48: [&lt;ffffffffa06e09e0&gt;] ll_file_io_generic+0x450/0x600 [lustre] 17:35:48: [&lt;ffffffffa06e1922&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] 17:35:48: [&lt;ffffffffa06e1c0c&gt;] ll_file_write+0x16c/0x2a0 [lustre] 17:35:48: [&lt;ffffffff81181398&gt;] vfs_write+0xb8/0x1a0 17:35:48: [&lt;ffffffff81181c91&gt;] sys_write+0x51/0x90 17:35:48: [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 17:35:48: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 17:35:48: 17:35:48:Kernel panic - not syncing: LBUG </pre> </div></div> <p>Patch back-ported to Lustre b2_4 branch: <a href="http://review.whamcloud.com/8530" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8530</a></p> <p>Patch landed on Lustre b2_4 branch for 2.4.2.</p> <p>Patch is being reverted from b2_4 due to test failures: <a href="http://review.whamcloud.com/8621" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8621</a></p> <p>We saw the same assertion in production at LLNL with Lustre 2.4.0-19chaos. Unfortunately it did not dump the backtrace to the console, and the crash dump failed. But the assertion and particular unknown state (4) were the same as in this ticket, so I am assuming that it is the same bug.</p> <p>Since the fix needed to be reverted, it would appear that this issue needs to be reopened.</p> <p>Christopher - I'm not sure what exactly your 2.4.0-19 consists of, but Intel's 2.4.0 does not have this patch reverted. Only if you're pulling updated sources from the b2.4 (or whatever it's called - sorry) branch in the repo, would you have the patch reverted.</p> <p>If you've got 2.4.0 (or even 2.4.1, I believe), you wouldn't have the patch to begin with.</p> <p>Patrick - I understand that. We hit the bug described in this ticket, not the bug caused by the fix for this ticket. We would like to be able to cherry-pick the fix so that our client does not assert hit this bug, but if the patch needed to be reverted it is not yet safe for me to do so.</p> <p>The fix of <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> has been landed on master &amp; b2_5, it looks to me that we should just re-add this patch and backport the fix of <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> to b2_4.</p> <p>Bob, what's your opinion? I see the revert patch was uploaded by you, why we revert this patch instead of backport the <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>?</p> <p>Just timing. At the time the fix for <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> was unproven so reverting to the previous known state was preferable. I think that moving forward with both fixes is a sound approach.</p> <p>ported <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a> fix to b2_4: <a href="http://review.whamcloud.com/9194" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9194</a></p> <p>Niu/Peter, will this backport be merged into b2_4 (for 2.4.4)? I saw this bug on three lustre 2.4.2 clients and AFAIKS your patch is not included in 2.4.3 (we're using this version now) or b2_4.</p> <p>Karsten</p> <p>This would certainly be under consideration for a fuller 2.4.x maintenance release. 2.4.3 had limited content because it was an unscheduled release driven by the need to issue a 2.4.x release to address the security vulnerability discovered (<a href="https://jira.whamcloud.com/browse/LU-4703" title="setxattr(2) will succeed by a non root user, against a file the user doesn&#39;t own." class="issue-link" data-issue-key="LU-4703"><del>LU-4703</del></a>/4704)</p> <p>Peter</p> Related LU-3889 LU-3020 LU-5910 Development Rank 1|hzvsov: Rank (Obsolete) 8552 Severity