[LU-4584] Lock revocation process fails consistently

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Labels: llnl
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

Some users have reported to us that the "rm" command is taking a long time. Some investigation revealed that at least the first "rm" in a directory takes just over 100 seconds, which of course sounds like OBD_TIMEOUT_DEFAULT.

This isn't necessarily the simplest reproducer, but the following reproducer is completely consistent:

1. set directory striping default count to 48
2. touch a file on client A
3. rm file on client B

The clients are running 2.4.0-19chaos, servers are at 2.4.0-21chaos. The servers are using zfs as the backend.

I have some lustre logs that I will share and talk about in additional posts to this ticket. But essentially it looks like the server always times out on a AST to client A (explaining the 100 second delay). It is not really clear yet to me why that happens, because client A appears to be completely responsive. My current suspicion is the the MDT is to blame.

Issue Links

• duplicates: LU-4963 client eviction during IOR test - lock callback timer expired
• is related to: LU-5525 ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed
• is related to: LU-5632 ldlm_lock_addref()) ASSERTION( lock != ((void *)0) )
• is related to: LU-5686 (mdt_handler.c:3203:mdt_intent_lock_replace()) ASSERTION( lustre_msg_get_flags(req->rq_reqmsg) & 0x0002 ) failed
• is related to: LU-2827 mdt_intent_fixup_resent() cannot find the proper lock in hash

Comments

[Andreas Dilger]

No updates in ticket since sites have upgraded to newer releases.

[James A Simmons]

I have been testing the b2_4 version of the LU-4584 patch and I see this:

LustreError: 4012:0:(ldlm_resource.c:1445:ldlm_resource_dump()) — Resource: [0xdeaf:0x0:0x0].0 (ffff88020290da80) refcount = 2

Its my simul job struggling.

[Christopher Morrone (Inactive)]

We won't know until it is installed on larger systems.

[James A Simmons]

Chris did you try the patch set for 2.4? If you did was their positive results?

[Oleg Drokin]

re 1.8 clients - since they don't know about wide striping, they hit this because their buffers are not large enough anyway, I suspect, when you have enough stripes.

The patches are needed regardless of course because the bugs are real during resends. It's just the resends used to be rare and only in the face of lost messages.

[James A Simmons]

Actually our Cray clients don't have the LU-3338 patch and the problems of LU-5530 still showed up. So these patches are still needed. We plan to do a full scale test shot Tuesday next week with all the above patches plus a few others. If it goes well we will leave this system running 2.5 clients and 2.5 servers.

We still need this work due to another file system running at the lab that will be stuck at 2.4 for the time being but some of our clients will be moving to 2.5 so these problems will show up.

[Oleg Drokin]

For -2827, we have:
LU-2827 itself http://review.whamcloud.com/5978 and http://review.whamcloud.com/#/c/10378
LU-5266 http://review.whamcloud.com/10903
LU-5496 http://review.whamcloud.com/11469 and http://review.whamcloud.com/#/c/11644
LU-5579 http://review.whamcloud.com/#/c/11839/
and the final nail in this bug LU-5530: http://review.whamcloud.com/#/c/11841/1

Some of those patches are not picking cleanly into b2_5, but I know James Simmons from ORNL ported them. All of this is now in testing after which they hopefully will publish their tree with backports.

Or another easy way to get rid of all of these problems (occuring as frequently, at least, sicne resend might also happen if a reply from server was genuinely lost on the network) is to drop LU-3338 patch, that is not part of standard b2_5 (only landed to b2_6)

[Christopher Morrone (Inactive)]

Ok, I will give those patches a try.

We are planning to try to move to b2_5 in less than 2 months, so we'll need the LU-2827 solution for b2_5 as soon as possible.

[Oleg Drokin]

patch http://review.whamcloud.com/#/c/9488/ fixed to eliminate the assertion in mdt_intent_lock_replace() now.
Also 2.4 deployments would need to carry patch http://review.whamcloud.com/6511 from LU-3428. This should address all related woes in b2_4. 2.6+ will be fixed by patches from LU-2827 and friends. As for 2.5 I think we'll still move with -2827 as the more generic solution.

[Christopher Morrone (Inactive)]

Yes, that was my belief. I would like Intel to enumerate the failure modes that users can expect to be fixed, and those that will not be fixed by LU-4584.