[LU-874] Client eviction on lock callback timeout

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.1
• Labels: llnl ptr
• Environment: https://github.com/chaos/lustre/tree/2.1.0-13chaos
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Christopher Morrone (Inactive)

Description

Our testing has revealed that lustre 2.1 is far more likely than 1.8 to return short reads and writes (return code says fewer bytes read/written than requested).

So far, the frequent reproducer is IOR shared single file, transfer size 128MB, block size 256MB, 32 client nodes, and 512 tasks evenly spread over the clients.

The file is only striped over 2 OSTs.

When the read() or write() return value is less than the requested amount, the size is, in every instance that I have seen thus far, a multiple of 1MB.

I suspect that other loads will show the same problem. I think that our more common large-transfer-request work loads come from our file-per-process apps though, so we'll run some tests to see if it is easy to reproduce there as well.

Issue Links

• is duplicated by: LU-2683 Client deadlock in cl_lock_mutex_get
• is duplicated by: LU-1065 High rate of obd_ping failure with client <-> OST evictions
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Jinshan Xiong (Inactive)]

Thank you Bruno, let's start the landing process then.

[Bruno Faccini (Inactive)]

Jinshan, we are now with 2 more days (total of more than 5 now !) of testing this latest patch on 2 different clusters running with multiple configurations, with heavy reproducer exposure. But still no hang reproduction.

[Bruno Faccini (Inactive)]

Not reproduced during the week-end, I keep trying using different Clients/Servers scenarios/configs.

[Jinshan Xiong (Inactive)]

Hi Bruno Faccini, can you please try to check if this works: http://review.whamcloud.com/5208

[Jinshan Xiong (Inactive)]

I will try to make a patch tomorrow.

[Oleg Drokin]

I am not working on this issue myself, I had a discussion with Jinshan yesterday and I believe we now agree the reason for this particular problem is understood.
Jinshan is now thinking on how to properly address it and patch 4306 might be the move in the right direction, but I'll defer to him.

The old rule of "ptlrpcd should not block" is still very important, when it's violated, all async ptlrpcd-handled RPCs are at risk of being stuck because their replies could not be handled.

[Bruno Faccini (Inactive)]

You are right I had to be quick and did not save the necessary symbol stuff, but at this moment it was not intended to serve after more than a month and also I thought all could be easily retrieved with their build versions (saved in job_hang.txt file) ... But if finally not, you are right that's a good experience to learn from for next time.

If you want I can reproduce it again on Toro ?

And according to your analysis we need to pursue with http://review.whamcloud.com/4306 ?

[Oleg Drokin]

Bruno, it would have been great if together with vmcore files you'd put lustre modules rpm and the kernel-debuginfo so that they are useful for everybody.

From my look it seems Bobijam analysus from Oct 16 is pretty much what I would think.

There's an extra piece here that explains it all. Reply for the RPC is not processed because ptlrpcd is locked up trying to get mutex, as the result, RPC (reply arrived based on flags) is stuck on the sending list and could not be taken out (also job for ptlrpcd that is locked up).

[Jinshan Xiong (Inactive)]

Thank you Bruno.

[Bruno Faccini (Inactive)]

Jinshan,

As I keep Toro reservations since quite a long time for this problem/reproducer, I need to free them now due to higher priority tests to be done on the cluster. So i decided to freeze the Nodes situations by 1st taking a "dump live" image of the 4 client-12vm[1-4] VMs with virsh and then I crashed all Nodes+VMs with Alt+SysRq sequence. All these infos are now available under brent:~bruno/LU-874/new_hang_101212.

[Bruno Faccini (Inactive)]

Sorry Doug, you may wait from some feedback here, so yes this are the msgs that we can see when we reproduce the hung situation, and the current hung thread stack on Client too.