[LU-2232] LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.2
• Labels: llnl
• Environment: Dell R710 servers running TOSS-2.0-2 and DDN 10k storage.
• Severity: 4
• Assignee: Lai Siyao
• Reporter: Joe Mervini

Description

Last night we had two OSSs panic at virtually the same time with and LBUG error being thrown. We just updated our servers and clients to 2.1.2-4chaos from 2.1.2-3chaos releases with the past 2 days and had not experienced this issue with the previous release. Below is a sample of the console log from one of the servers. I have also captured all the console messages up until the system panicked and am attaching it.

LustreError: 9044:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG
Pid: 9120, comm: ll_ost_io_341

Call Trace:
LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG
[<ffffffffa0440895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Pid: 9083, comm: ll_ost_io_304

Issue Links

• is duplicated by: LU-4844 ost_prolong_lock_one()) ASSERTION( lock->l_export == opd->opd_exp )
• is related to: LU-5116 Race between resend and reply processing

Comments

[Peter Jones]

Thanks Marc!

[D. Marc Stearman (Inactive)]

We have not seen any more occurrences of this error since we rolled out our 2.5.4-4chaos version into production.

[Christopher Morrone (Inactive)]

I have pulled change 14950, Patch Set 1, into LLNL's local tree. It is in the queue to go into the next TOSS release and eventually roll out into production.

[Lai Siyao]

The updated debug patch will print both lock and request details, which can tell us whether this request is new, or resent/replay request.

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14950
Subject: LU-2232 debug: print debug for prolonged lock
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 1d21596de85b81bb3f98277f8f0b425368ccb187

[Lai Siyao]

The log shows after client eviction (and quite possibly reconnected), a stale lock handle is packed in client rw request.

I added some code to try to simulate this, but failed. According to the code, after eviction, client inflight RPCs will be aborted, and locks be cleaned up. If a full debug log covering this recovery(at least on client) can be obtained upon this error message is seen, it can help move this forward.