[LU-5910] LustreError: osc_lock_enqueue() Impossible state: 4

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.3
• Type: Bug
• Priority: Minor
• Reporter: Jay Lan (Inactive)
• Assignee: Zhenyu Xu
• Resolution: Cannot Reproduce

Environment

sles11sp3 3.0.101-0.31.1
Front end client node
lustre 2.4.3-7nasC, source at https://github.com/jlan/lustre-nas/tree/nas-2.4.3

• Severity: 3

Description

Pfe20, a front end client node, dropped to KDB on Nov 10, 2014.

Console log showed:

CONSOLE MARK 201411100930
[-- MARK -- Mon Nov 10 10:00:00 2014]^M
[605803.620800] LustreError: 82374:0:(osc_lock.c:1165:osc_lock_enqueue()) ASSERTION( ols->ols_state == OLS_NEW ) failed: Impossible state: 4^M
[605803.657700] LustreError: 82374:0:(osc_lock.c:1165:osc_lock_enqueue()) LBUG^M
[605803.678499] Pid: 82374, comm: scp^M
[605803.688669] ^M
[605803.688670] Call Trace:^M
[605803.700978] [<ffffffff81004935>] dump_trace+0x75/0x310^M
^M
Entering kdb (current=0xffff88095519e480, pid 82374) on processor 3 Oops: (null)^M
due to oops @ 0x0^M
kdba_dumpregs: pt_regs not available, use bt* or pid to select a different task^M
[3]kdb>

Call trace:
<0>[605803.701897] Kernel panic - not syncing: LBUG^M
<4>[605803.701900] Pid: 82374, comm: scp Tainted: GF NX 3.0.101-0.31.1.20140612-nasa #1^M
<4>[605803.701901] Call Trace:^M
<4>[605803.701906] [<ffffffff81004935>] dump_trace+0x75/0x310^M
<4>[605803.701911] [<ffffffff8146ef53>] dump_stack+0x69/0x6f^M
<4>[605803.701914] [<ffffffff8146efec>] panic+0x93/0x201^M
<4>[605803.701923] [<ffffffffa075fdc3>] lbug_with_loc+0xa3/0xb0 [libcfs]^M
<4>[605803.701935] [<ffffffffa0b91581>] osc_lock_enqueue+0x3f1/0x4c0 [osc]^M
<4>[605803.701963] [<ffffffffa08c4722>] cl_enqueue_kick+0x62/0x150 [obdclass]^M
<4>[605803.702008] [<ffffffffa08c7f31>] cl_enqueue_try+0xc1/0x210 [obdclass]^M
<4>[605803.702073] [<ffffffffa0c27e2d>] lov_lock_enqueue_one+0x4d/0x170 [lov]^M
<4>[605803.702092] [<ffffffffa0c2b07b>] lov_lock_enqueue+0x16b/0x3e0 [lov]^M
<4>[605803.702122] [<ffffffffa08c4722>] cl_enqueue_kick+0x62/0x150 [obdclass]^M
<4>[605803.702167] [<ffffffffa08c7f31>] cl_enqueue_try+0xc1/0x210 [obdclass]^M
<4>[605803.702213] [<ffffffffa08c8817>] cl_enqueue_locked+0x77/0x1e0 [obdclass]^M
<4>[605803.702259] [<ffffffffa08c9509>] cl_lock_request+0x79/0x280 [obdclass]^M
<4>[605803.702305] [<ffffffffa08ce673>] cl_lockset_lock_one+0x53/0x190 [obdclass]^M
<4>[605803.702353] [<ffffffffa08ceac1>] cl_lockset_lock+0x51/0x1b0 [obdclass]^M
<4>[605803.702401] [<ffffffffa08cecf8>] cl_io_lock+0xd8/0x150 [obdclass]^M
<4>[605803.702449] [<ffffffffa08cee02>] cl_io_loop+0x92/0x190 [obdclass]^M
<4>[605803.702491] [<ffffffffa0cadc72>] ll_file_io_generic+0x342/0x600 [lustre]^M
<4>[605803.702507] [<ffffffffa0cae3f3>] ll_file_aio_read+0x233/0x290 [lustre]^M
<4>[605803.702523] [<ffffffffa0caef03>] ll_file_read+0x203/0x290 [lustre]^M
<4>[605803.702530] [<ffffffff8115b2c7>] vfs_read+0xc7/0x130^M
<4>[605803.702533] [<ffffffff8115b433>] sys_read+0x53/0xa0^M
<4>[605803.702537] [<ffffffff81479c92>] system_call_fastpath+0x16/0x1b^M
<4>[605803.702544] [<00007fffec9b43c0>] 0x7fffec9b43bf^M

Issue Links

• is related to: LU-4591 Related cl_lock failures on master/2.5
• is related to: LU-3433 Encountered a assertion for the ols_s...
• is related to: LU-5599 Lustre Error: Impossible state: 4

Comments

[Peter Jones]

Bobijam

Could you please advise?

Thanks

Peter

[Zhenyu Xu]

do you have these 2 patches ?
http://review.whamcloud.com/#/c/8530/
http://review.whamcloud.com/#/c/9783/

[Jay Lan (Inactive)]

We have #9783. As to #8530, it was reverted before 2.4.2 release, isn't it?

[Zhenyu Xu]

yes, it was reverted. But as time goes, #9194 is the cure, and if #9194 is in place, #8530 can/should be revived.

[Jay Lan (Inactive)]

We had another front end client node hitting this bug this morning.

We already have #9194. So, do you think #8530 may solve our problem?

The problem is we do not know how to reproduce the bug. It has not happened for a few months and suddenly we hit this bug twice in span of about a week...

Just checked. #8530 was not reverted in b2_6 nor in master. OK, I will put it back in to our nas-2.4.3.

[Jay Lan (Inactive)]

We have not seen this for a long while and we are running 2.7, so please close it.

[Peter Jones]

ok Jay