[LU-2232] LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.2
• Labels: llnl
• Environment: Dell R710 servers running TOSS-2.0-2 and DDN 10k storage.
• Severity: 4
• Assignee: Lai Siyao
• Reporter: Joe Mervini

Description

Last night we had two OSSs panic at virtually the same time with and LBUG error being thrown. We just updated our servers and clients to 2.1.2-4chaos from 2.1.2-3chaos releases with the past 2 days and had not experienced this issue with the previous release. Below is a sample of the console log from one of the servers. I have also captured all the console messages up until the system panicked and am attaching it.

LustreError: 9044:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9120:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG
Pid: 9120, comm: ll_ost_io_341

Call Trace:
LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) ASSERTION(lock->l_export == opd->opd_exp) failed
LustreError: 9083:0:(ost_handler.c:1673:ost_prolong_lock_one()) LBUG
[<ffffffffa0440895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Pid: 9083, comm: ll_ost_io_304

Issue Links

• is duplicated by: LU-4844 ost_prolong_lock_one()) ASSERTION( lock->l_export == opd->opd_exp )
• is related to: LU-5116 Race between resend and reply processing

Comments

[Patrick Farrell (Inactive)]

Lai - Given that the code in this area was changed a fair bit in 2.6, do you think this bug is likely to still be present?

[Peter Jones]

thanks Patrick. Wow. That is odd.

[Patrick Farrell (Inactive)]

Peter,

That's correct. It was on b2_5.

• Patrick

[Peter Jones]

Patrick

To be clear, Cray could reliably reproduce this issue, then applied the diagnostic patch and could not? What code line was this on?

Thanks

Peter

[Patrick Farrell (Inactive)]

Lai - Sorry we didn't update this earlier. Cray tried to reproduce this bug with your debug patch, and were unable to do so. After that, we pulled your patch set 1 in to our Lustre version and haven't seen this bug since.

[Peter Jones]

It would be a sensible approach to try both the fix from LU-5116 and the diagnostic patch from LU-2232. That way perhaps the issue would be fixed by LU-5116 but if there is still a residual problem we will have fuller information to go forward on.

[Oleg Drokin]

I think LU-5116 might be related here to explan resend across eviction

[Lai Siyao]

I did some test but couldn't reproduce this failure, so I updated the patch for 2.4/2.5 to debug patch which will print lock and request export address and crash as before, so that we can dump these two exports to help analyse.

Could you apply the patch and reproduce it? and if it crashes upload the crash dump.

[Lai Siyao]

I'm afraid some requests were not aborted upon eviction on client, so that they were resent through new connection. I'll make some test to find more proof.

[Oleg Drokin]

Question - if the client was evicted - the request resending should have failed because the server would reject it.
So there's something else happening that your explanation does not quite explain I think.

[Lai Siyao]

Patches are ready:
2.4: http://review.whamcloud.com/#/c/9925/
2.5: http://review.whamcloud.com/#/c/9926/
master: http://review.whamcloud.com/#/c/9927/