[LU-4584] Lock revocation process fails consistently

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Labels: llnl
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

Some users have reported to us that the "rm" command is taking a long time. Some investigation revealed that at least the first "rm" in a directory takes just over 100 seconds, which of course sounds like OBD_TIMEOUT_DEFAULT.

This isn't necessarily the simplest reproducer, but the following reproducer is completely consistent:

1. set directory striping default count to 48
2. touch a file on client A
3. rm file on client B

The clients are running 2.4.0-19chaos, servers are at 2.4.0-21chaos. The servers are using zfs as the backend.

I have some lustre logs that I will share and talk about in additional posts to this ticket. But essentially it looks like the server always times out on a AST to client A (explaining the 100 second delay). It is not really clear yet to me why that happens, because client A appears to be completely responsive. My current suspicion is the the MDT is to blame.

Issue Links

• duplicates: LU-4963 client eviction during IOR test - lock callback timer expired
• is related to: LU-5525 ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed
• is related to: LU-5632 ldlm_lock_addref()) ASSERTION( lock != ((void *)0) )
• is related to: LU-5686 (mdt_handler.c:3203:mdt_intent_lock_replace()) ASSERTION( lustre_msg_get_flags(req->rq_reqmsg) & 0x0002 ) failed
• is related to: LU-2827 mdt_intent_fixup_resent() cannot find the proper lock in hash

Comments

[Oleg Drokin]

For -2827, we have:
LU-2827 itself http://review.whamcloud.com/5978 and http://review.whamcloud.com/#/c/10378
LU-5266 http://review.whamcloud.com/10903
LU-5496 http://review.whamcloud.com/11469 and http://review.whamcloud.com/#/c/11644
LU-5579 http://review.whamcloud.com/#/c/11839/
and the final nail in this bug LU-5530: http://review.whamcloud.com/#/c/11841/1

Some of those patches are not picking cleanly into b2_5, but I know James Simmons from ORNL ported them. All of this is now in testing after which they hopefully will publish their tree with backports.

Or another easy way to get rid of all of these problems (occuring as frequently, at least, sicne resend might also happen if a reply from server was genuinely lost on the network) is to drop LU-3338 patch, that is not part of standard b2_5 (only landed to b2_6)

[Christopher Morrone (Inactive)]

Ok, I will give those patches a try.

We are planning to try to move to b2_5 in less than 2 months, so we'll need the LU-2827 solution for b2_5 as soon as possible.

[Oleg Drokin]

patch http://review.whamcloud.com/#/c/9488/ fixed to eliminate the assertion in mdt_intent_lock_replace() now.
Also 2.4 deployments would need to carry patch http://review.whamcloud.com/6511 from LU-3428. This should address all related woes in b2_4. 2.6+ will be fixed by patches from LU-2827 and friends. As for 2.5 I think we'll still move with -2827 as the more generic solution.

[Christopher Morrone (Inactive)]

Yes, that was my belief. I would like Intel to enumerate the failure modes that users can expect to be fixed, and those that will not be fixed by LU-4584.

[James A Simmons]

I have several different reproducers of this problem. What I found what LU-4584 address some of the reproducers but not all of them. The patch for LU-2827 addressed more of my reproducers but I still had client evictions.

[Christopher Morrone (Inactive)]

Could you please provide an explanation of what operations will not be fixed by the LU-4584 patch, as compared with the more general LU-2827 fix?

[Christopher Morrone (Inactive)]

FYI, we put the LU-4584 patch into production, and it didn't take too long before we hit the assertion that James reported. I opened LU-5525 for that bug.

I am beginning to suspect that the patch either did cause the assertion, or made the assertion more common.

[James A Simmons]

I have been testing with the LU-4584 patch and I'm still seeing client evictions. Could it be possible to get the LU-2827 patch working on 2.4

[James A Simmons]

It was my bad. The last test shot we used our 2.4 production file system which didn't have the patch from here. So the above breakage is expected. We are in the process of testing this at larger scale (500 nodes) production machine. Yes ORNL has created a public git tree

https://github.com/ORNL-TechInt/lustre

So people can examine our special sauce.

[Christopher Morrone (Inactive)]

James, can you share the patch stack you are using? That might help us figure out if you are reporting the same issue or something else. And if it isn't exactly the same issue, we really need to get you to report it in another ticket.

[James A Simmons]

Just finished a test shot with Cray 2.5 clients to see if the client evicts stopped. Their default client which is some 2.5 version with many many patches lacked the LU-2827 and LU-4861 patches I founded that helped with 2.5.2. So I applied patches from LU-2827 and LU-4861 but still had client evicts. I collected the logs from the server side and have placed them here:

ftp.whamcloud.com/uploads/LU-4584/atlas2_testshot_Jul_29_2014_debug_logs.tar.gz