[LU-10463] Poor write performance periodically on repeated test runs

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.11.0, Lustre 2.10.2
• Environment: Centos 7.4, various Lustre and ZFS versions tested. Lustre clients are 2.10.2_RC2.
• Severity: 3
• Assignee: Peter Jones
• Reporter: Andreas Dilger

Description

I'm running an IOR test (IOR-2.10.3) that writes 1GB files to one dataset/directory, then writes 3GB files to another dataset/directory, then reads back the first dataset. This test sequence is run 25 times. My filesystem is able to do 14-16GB/sec writes, and most iterations of this test will produce that bandwidth. Problem is that out of the 25 iterations, a couple/few of the test iterations turn in significantly lower results often in the 5-10GB/sec range.

I initially suspected hardware issues, but testing of components including each individual disk drive showed everything working properly, and I've seen nothing in the logs when running the test above reporting any problem. So, I started building and testing various combinations of Lustre and ZFS. The hardware, clients and server OS have been constant for each of the tests. Only SPL/ZFS and Lustre on the server have changed from test to test.

It appears to boil down to the problem having been introduced in the Lustre 2.10.x branch. I have not seen the problem occur in the Lustre 2.9 builds I've done. I've built Lustre 2.9 with ZFS 0.7.3 and seen no issue. I've build Lustre 2.10.x with ZFS 0.6.5.7 and do observe the issue. Every build I've done with Lustre 2.10.x (several) showed the issue.

Issue Links

• is related to: LU-10465 increase default stripe size to 4MB
• is related to: LU-18635 add non-page tunable max_mb_per_rpc_read/write parameters
• is related to: LU-9090 increase default RPC size to 4MB

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/30969/
Subject: LU-10463 osd-zfs: use 1MB RPC size by default
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: f119ec3196eb3e7773eeb4dcb3d825d7f8725a9c

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30969
Subject: LU-10463 osd-zfs: use 1MB RPC size by default
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 79f3e1a4fa0ed94ee3958c955471d3ba67050a60

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30757/
Subject: LU-10463 osd-zfs: use 1MB RPC size by default
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: af34a876d2ebde2b4717c920683c7fc8b5eae1cf