[LU-874] Client eviction on lock callback timeout

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.1
• Labels: llnl ptr
• Environment: https://github.com/chaos/lustre/tree/2.1.0-13chaos
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Christopher Morrone (Inactive)

Description

Our testing has revealed that lustre 2.1 is far more likely than 1.8 to return short reads and writes (return code says fewer bytes read/written than requested).

So far, the frequent reproducer is IOR shared single file, transfer size 128MB, block size 256MB, 32 client nodes, and 512 tasks evenly spread over the clients.

The file is only striped over 2 OSTs.

When the read() or write() return value is less than the requested amount, the size is, in every instance that I have seen thus far, a multiple of 1MB.

I suspect that other loads will show the same problem. I think that our more common large-transfer-request work loads come from our file-per-process apps though, so we'll run some tests to see if it is easy to reproduce there as well.

Issue Links

• is duplicated by: LU-2683 Client deadlock in cl_lock_mutex_get
• is duplicated by: LU-1065 High rate of obd_ping failure with client <-> OST evictions
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Stephen Champion]

I noticed that http://review.whamcloud.com/5208 was committed to master but not to b2_1, while all of the other patches committed for LU-874 have been applied to both trees. Oversight?

[Jinshan Xiong (Inactive)]

Thank you Bruno. Let's land 5208 in this case.

[Bruno Faccini (Inactive)]

Jinshan, I post this here since I think it belongs to this JIRA !!
So again, no more occurence with reproducer and also others tests/work-loads when running with http://review.whamcloud.com/5208 patch #2, since more than 2 full days now.

[Jodi Levi (Inactive)]

LU-2683 is the release blocker.

[Jinshan Xiong (Inactive)]

Thank you Bruno, let's start the landing process then.

[Bruno Faccini (Inactive)]

Jinshan, we are now with 2 more days (total of more than 5 now !) of testing this latest patch on 2 different clusters running with multiple configurations, with heavy reproducer exposure. But still no hang reproduction.

[Bruno Faccini (Inactive)]

Not reproduced during the week-end, I keep trying using different Clients/Servers scenarios/configs.

[Jinshan Xiong (Inactive)]

Hi Bruno Faccini, can you please try to check if this works: http://review.whamcloud.com/5208

[Jinshan Xiong (Inactive)]

I will try to make a patch tomorrow.

[Oleg Drokin]

I am not working on this issue myself, I had a discussion with Jinshan yesterday and I believe we now agree the reason for this particular problem is understood.
Jinshan is now thinking on how to properly address it and patch 4306 might be the move in the right direction, but I'll defer to him.

The old rule of "ptlrpcd should not block" is still very important, when it's violated, all async ptlrpcd-handled RPCs are at risk of being stuck because their replies could not be handled.

[Bruno Faccini (Inactive)]

You are right I had to be quick and did not save the necessary symbol stuff, but at this moment it was not intended to serve after more than a month and also I thought all could be easily retrieved with their build versions (saved in job_hang.txt file) ... But if finally not, you are right that's a good experience to learn from for next time.

If you want I can reproduce it again on Toro ?

And according to your analysis we need to pursue with http://review.whamcloud.com/4306 ?