[LU-9574] Large file read performance degradation from multiple OST's

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.2
• Affects Version/s: Lustre 2.9.0
• Environment: RHEL 7 servers, RHEL 6 and 7 clients.

Epic/Theme

Performance

• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Darby Vicker

Description

We recently noticed that the large file read performance on our 2.9 LFS is dramatically worse than it used to be. The attached plot is the result of a test script that uses dd to write a large file (50GB) to disk, read that file and then copy it to a 2nd file to test write, read and read/write speeds for large files for various stripe sizes and counts. The two sets of data on this plot are on the same server and client hardware. The LFS was originally built and formatted with 2.8.0 but we eventually upgraded to 2.9.0 on the servers and clients. The behavior we are used to seeing is increasing performance as you increase the stripe count with a peak in performance around 4 or 6 OST's and a degradation after that as more OST's are used. This is what we saw under 2.8 (red lines in the plots). With 2.9 we still get very good write performance (almost line rate on our 10 GbE clients). But for reads we see extremely good performance with a single OST and significantly degraded performance for multiple OST's – black lines in the plots.  Using a git bisect to compile and test different clients, we were able to isolate it to this commit:

commit d8467ab8a2ca15fbbd5be3429c9cf9ceb0fa78b8
LU-7990 clio: revise readahead to support 16MB IO

There is slightly more info here:

http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2017-May/014509.html

Please let me know if you need any other data or info.   

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/29016/
Subject: LU-9574 llite: pipeline readahead better with large I/O
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 73a7af47a01979d5dd5c7e8dcaf3f62d47f11b0b

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27388/
Subject: LU-9574 llite: pipeline readahead better with large I/O
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 011742134e3152f3e389ec30c08ccfc28d7a91a7

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29016
Subject: LU-9574 llite: pipeline readahead better with large I/O
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 83946d6f389a5c047cc958973c4b2cdf291a6429