[LU-874] Client eviction on lock callback timeout

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.2.0, Lustre 2.4.0, Lustre 2.1.1
• Labels: llnl ptr
• Environment: https://github.com/chaos/lustre/tree/2.1.0-13chaos
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Christopher Morrone (Inactive)

Description

Our testing has revealed that lustre 2.1 is far more likely than 1.8 to return short reads and writes (return code says fewer bytes read/written than requested).

So far, the frequent reproducer is IOR shared single file, transfer size 128MB, block size 256MB, 32 client nodes, and 512 tasks evenly spread over the clients.

The file is only striped over 2 OSTs.

When the read() or write() return value is less than the requested amount, the size is, in every instance that I have seen thus far, a multiple of 1MB.

I suspect that other loads will show the same problem. I think that our more common large-transfer-request work loads come from our file-per-process apps though, so we'll run some tests to see if it is easy to reproduce there as well.

Issue Links

• is duplicated by: LU-2683 Client deadlock in cl_lock_mutex_get
• is duplicated by: LU-1065 High rate of obd_ping failure with client <-> OST evictions
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[James A Simmons]

Can we close this please

[Bruno Faccini (Inactive)]

Stephen, I just back-ported Change #5208 (patch-set #2) as part of LU-2683.

[Stephen Champion]

I noticed that http://review.whamcloud.com/5208 was committed to master but not to b2_1, while all of the other patches committed for LU-874 have been applied to both trees. Oversight?

[Jinshan Xiong (Inactive)]

Thank you Bruno. Let's land 5208 in this case.

[Bruno Faccini (Inactive)]

Jinshan, I post this here since I think it belongs to this JIRA !!
So again, no more occurence with reproducer and also others tests/work-loads when running with http://review.whamcloud.com/5208 patch #2, since more than 2 full days now.

[Jodi Levi (Inactive)]

LU-2683 is the release blocker.

[Jinshan Xiong (Inactive)]

Thank you Bruno, let's start the landing process then.

[Bruno Faccini (Inactive)]

Jinshan, we are now with 2 more days (total of more than 5 now !) of testing this latest patch on 2 different clusters running with multiple configurations, with heavy reproducer exposure. But still no hang reproduction.

[Bruno Faccini (Inactive)]

Not reproduced during the week-end, I keep trying using different Clients/Servers scenarios/configs.

[Jinshan Xiong (Inactive)]

Hi Bruno Faccini, can you please try to check if this works: http://review.whamcloud.com/5208