[LU-3433] Encountered a assertion for the ols_state being set to a impossible state

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.5.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1
• Labels: mn4
• Environment: Cray Lustre 2.4 clients running on SLES11 SP2.
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: James A Simmons

Description

During a test run using Lustre 2.4 one of our clients encountered this LBUG.

[2013-05-29 18:39:47][c7-1c1s3n3]LustreError:16573:0:(osc_lock.c:1165:osc_lock_enqueue()) ASSERTION( ols->ols_state == OLS_NEW ) failed: Impossible state: 4
[2013-05-29 18:39:47][c7-1c1s3n3]LustreError: 16573:0
(osc_lock.c:1165:osc_lock_enqueue()) LBUG
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff81006451>] try_stack_unwind+0x161/0x1a0
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff81004ca9>] dump_trace+0x89/0x440
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa013a897>] libcfs_debug_dumpstack+0x57/0x80 [libcfs]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa013ade7>] lbug_with_loc+0x47/0xc0 [libcfs]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa0639f55>] osc_lock_enqueue+0x725/0x8b0 [osc]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa032d4eb>] cl_enqueue_try+0xfb/0x320 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa06ccded>] lov_lock_enqueue+0x1fd/0x880 [lov]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa032d4eb>] cl_enqueue_try+0xfb/0x320 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa032e3bf>] cl_enqueue_locked+0x7f/0x1f0 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa032efbe>] cl_lock_request+0x7e/0x270 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa0334274>] cl_io_lock+0x394/0x5c0 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa033453a>] cl_io_loop+0x9a/0x1a0 [obdclass]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa074b90f>] ll_file_io_generic+0x33f/0x5f0 [lustre]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa074c08b>] ll_file_aio_read+0x23b/0x290 [lustre]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffffa074d002>] ll_file_read+0x1f2/0x280 [lustre]
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff81135548>] vfs_read+0xc8/0x180
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff8113b799>] kernel_read+0x49/0x60
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff8113b885>] prepare_binprm+0xd5/0x100
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff8113c670>] do_execve_common+0x1c0/0x300
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff8113c83f>] do_execve+0x3f/0x50
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff8100ae1e>] sys_execve+0x4e/0x80
[2013-05-29 18:39:47][c7-1c1s3n3] [<ffffffff81316a3c>] stub_execve+0x6c/0xc0
[2013-05-29 18:39:47][c7-1c1s3n3] [<0000000020176437>] 0x20176437
[2013-05-29 18:39:47][c7-1c1s3n3]Kernel panic - not syncing: LBUG

Issue Links

• is related to: LU-5910 LustreError: osc_lock_enqueue() Impossible state: 4
• is related to: LU-3889 LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock->cll_state >= CLS_QUEUING )
• is related to: LU-3020 Lustre returns EINTR during writes when SA_RESTART is set

Comments

[Peter Jones]

Karsten

This would certainly be under consideration for a fuller 2.4.x maintenance release. 2.4.3 had limited content because it was an unscheduled release driven by the need to issue a 2.4.x release to address the security vulnerability discovered (LU-4703/4704)

Peter

[Karsten Weiss]

Niu/Peter, will this backport be merged into b2_4 (for 2.4.4)? I saw this bug on three lustre 2.4.2 clients and AFAIKS your patch is not included in 2.4.3 (we're using this version now) or b2_4.

[Niu Yawei (Inactive)]

ported LU-3889 fix to b2_4: http://review.whamcloud.com/9194

[Peter Jones]

Just timing. At the time the fix for LU-3889 was unproven so reverting to the previous known state was preferable. I think that moving forward with both fixes is a sound approach.

[Niu Yawei (Inactive)]

The fix of LU-3889 has been landed on master & b2_5, it looks to me that we should just re-add this patch and backport the fix of LU-3889 to b2_4.

Bob, what's your opinion? I see the revert patch was uploaded by you, why we revert this patch instead of backport the LU-3889?

[Christopher Morrone (Inactive)]

Patrick - I understand that. We hit the bug described in this ticket, not the bug caused by the fix for this ticket. We would like to be able to cherry-pick the fix so that our client does not assert hit this bug, but if the patch needed to be reverted it is not yet safe for me to do so.

[Patrick Farrell (Inactive)]

Christopher - I'm not sure what exactly your 2.4.0-19 consists of, but Intel's 2.4.0 does not have this patch reverted. Only if you're pulling updated sources from the b2.4 (or whatever it's called - sorry) branch in the repo, would you have the patch reverted.

If you've got 2.4.0 (or even 2.4.1, I believe), you wouldn't have the patch to begin with.

[Christopher Morrone (Inactive)]

We saw the same assertion in production at LLNL with Lustre 2.4.0-19chaos. Unfortunately it did not dump the backtrace to the console, and the crash dump failed. But the assertion and particular unknown state (4) were the same as in this ticket, so I am assuming that it is the same bug.

Since the fix needed to be reverted, it would appear that this issue needs to be reopened.

[Andreas Dilger]

Patch is being reverted from b2_4 due to test failures: http://review.whamcloud.com/8621

[Jian Yu]

Patch landed on Lustre b2_4 branch for 2.4.2.