[LU-12043] improve Lustre single thread read performances

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.14.0
• Type: Improvement
• Priority: Minor
• Reporter: Wang Shilong (Inactive)
• Assignee: Wang Shilong (Inactive)
• Resolution: Fixed
• Labels: LTS12

Description

There are several efforts here and there which try to improve the performances of single
thread read performances.

this ticket is opened to track a simple enough patch to improve the performances as much
as possible.

Here is whole history:

Currently, for sequential read IO, We grow up window
size very quickly, and once we cached @max_readahead_per_file
pages. For following command:

dd if=/mnt/lustre/file of=/dev/null bs=1M

We will do something like following:
...
64M bytes cached.
fast io for 16M bytes
readahead extra 16M to fill up window.
fast io for 16M bytes
readahead extra 16M to fill up window.
....

In this way, we could only use fast IO for 16M bytes and
then fall through non-fast IO mode. this is also reason
that why increasing @max_readahead_per_file don't give us
performances up, since this value only changes how much
memory we cached in memory, during my testing whatever
I changed the value, i could only get 2GB/s for single thread
read.

Actually, we could do this better, if we have used
more than 16M bytes readahead pages, submit another readahead
requests in the background. and ideally, we could always
use fast IO..I did a quick test with fake-IO on my limited
PC server:

Without patch VS Patched:
~2.0GB/S vs ~3.0 GB/s

So we could gain at least 50% performance up, i supposed
We could get more maybe more than 4GB/S with patch.

Issue Links

• is duplicated by: LU-10557 Should form read RPCs according to re...
• is related to: LU-8964 use parallel I/O to improve performan...
• is related to: LU-8709 parallel asynchronous readahead
• is related to: LU-12518 improve Lustre unaligned IO read perf...
• is related to: LU-13386 allow over reservation to cover curre...
• is related to: LU-11526 Support 64MB I/O RPC
• is related to: LU-13258 Bind linux workqueues to specific core

Comments

[Wang Shilong (Inactive)]

Patch is here.

https://review.whamcloud.com/#/c/34095/

[Wang Shilong (Inactive)]

some performances numbers:

We could get more than 3GB/s using default 64M redahead size.
After changing it to 128M, we could gain more than 4GB/s

So it is 100% performances up compared to without patch for the peak numbers.

[Shuichi Ihara]

Here is test resutls with/without patch.

Enviroment

1 x Client(1 x Platinum 8160 CPU @ 2.10GHz, 96GB DDR4 2667MHz memory, 1 x EDR Infiniband)
Mellanox OFED-4.5
CentOS7.6 (3.10.0-957.el7.x86_64)
1 x SFA200NVE(20 x NVMe) for OSS/OST

Test steps

# lctl set_param osc.*.max_pages_per_rpc=16M osc.*.max_rpcs_in_flight=16 osc.*.max_dirty_mb=512 llite.*.max_read_ahead_mb=2048 llite.*.max_read_ahead_per_file_mb=512
# mkdir /cache1/dir
# mkdir /cache1/sdir
# lfs setstripe -c -1 -S 16M /cache1/sdir
# mpirun -np 96 ior -w -t 1m -b 8g -e -vv -o /cache1/sdir/file -k
# mpirun -np 96 ior -w -t 1m -b 8g -e -vv -F -o /cache1/dir/file -k -F

# pdsh -w es400nv-vm[1-4],s184  "echo 3 > /proc/sys/vm/drop_caches"
# mpirun -np 1 ior -r -t 1m -b 768g -e -vv -o /cache1/sdir/file -k
# pdsh -w es400nv-vm[1-4],s184  "echo 3 > /proc/sys/vm/drop_caches"
# mpirun -np 96 ior -r -t 1m -b 8g -e -vv -o /cache1/dir/file -k -F
NOTE: chnage np value to 1, 24, 48 and 96. also transfer size is also teated with 4k and 1m.

Test results

NP TYPE     IOSIZE  master(MB/s) master+patch-34095(MB/s)                           
 1 SSF      4KB      1038         1618 <--- +60%
24 SSF      4KB      3134         2931
48 SSF      4KB      3020         2907
96 SSF      4KB      2746         2842

 1 SSF      1MB      1577         3040 <--- +100%
24 SSF      1MB      3285         3065
48 SSF      1MB      3124         3053
96 SSF      1MB      2820         2973

24 FPP      4KB      6496         7468
48 FPP      4KB      8306         8320
96 FPP      4KB      8365         9897

24 FPP      1MB      7744         7548
48 FPP      1MB      8874         8880
96 FPP      1MB      9907         9477

NOTE: multiple thread tests are just making sure no regression.

[Wang Shilong (Inactive)]

Thanks for Ihara's testing results!

[James A Simmons]

Peter I was talking to Wang about making this code CPT aware to limit noise on the client nodes. The question is do I create the patch for this under this ticket or create a new ticket?

[Peter Jones]

James

I suggest a new ticket linked to this one - it'll be easiest to track that way

Peter

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34602
Subject: LU-12043 llite: switch to use ll_fsname directly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f21f5d21e4ec0a0c4a3e232c6e2606cab8893142

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34755
Subject: LU-12043 llite, readahead: fix to call ll_ras_enter() properly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6e270c46b44aebbfacf7786c8c21736bc2bcecbf

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34602/
Subject: LU-12043 llite: switch to use ll_fsname directly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 506b68a359045cd8c78e5385eab6d4e01f2fdb83

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34755/
Subject: LU-12043 llite, readahead: fix to call ll_ras_enter() properly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 500edcada7e4101a21a28078611597fdb50b4791

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34849
Subject: LU-12043 llite: move readahead tunables to sysfs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9645f28c52be54e5f59516fdea2669cbd43d7061

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35033
Subject: LU-12043 llite,readahead: don't use max RPC size always
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eb9b7202da4b65d3d38edb53014b026bf8b2cfed

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35215
Subject: LU-12043 llite: make sure readahead cover current read
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 45c5046afe1ab8ff9fab71ff2b21b695651d7a73

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35216
Subject: LU-12043 llite: don't miss every first stride off page
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b81b34e2ca6188b7f579cdb53e12e3fed621527f

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35217
Subject: LU-12043 llite: fix bad performance with readahead disabled
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bb00859c21a79ce0d071e82519fe1f9aba94bce8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35033/
Subject: LU-12043 llite,readahead: don't always use max RPC size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7864a6854c3dfe3319dcf6809e728eed9a37b9b4

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34095/
Subject: LU-12043 llite: improve single-thread read performance
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c2791674260b120c596885c0356f33852cd7a685

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35438
Subject: LU-12043 llite: extend readahead locks for striped file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6b87ba4e55e44ad538bc047c279c856804b7144a

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35215/
Subject: LU-12043 llite: make sure readahead cover current read
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8fbef5ee761920faa3c84f142f56934d6266a409

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35559
Subject: LU-12043 llite,readahead: don't always use max RPC size
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: eccd7b3873e5395fc4087dec6b09acb91cadc1ef

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35216/
Subject: LU-12043 llite: don't miss every first stride page
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 29d8eb5ee7dff422294c9ee3d0f34f1370cec800

[Patrick Farrell (Inactive)]

Are there any patches still not landed under this ticket, wshilong ?

[Wang Shilong (Inactive)]

Yup, still two patches not landed yet.

https://review.whamcloud.com/35217
LU-12043 llite: fix bad performance with readahead disabled

https://review.whamcloud.com/35438
LU-12043 llite: extend readahead locks for striped file

I think second one will be landed soon, while the first one need be some refreshed.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35438/
Subject: LU-12043 llite: extend readahead locks for striped file
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cfbeae97d736152e5bdf8f3e759a260f6ee903d8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35559/
Subject: LU-12043 llite,readahead: don't always use max RPC size
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 57d2dd820ed430009a6146180d11c5e2b04b5426

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37620
Subject: LU-12043 llite: move tunable params to sysfs_memparse()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9fa91e61e5daccc9bba90b9c65c9fb054715b778

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34849/
Subject: LU-12043 llite: move tunable params to sysfs_memparse()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 59a3d7237df82ac2ce14b6779577584163f9122b

[Wang Shilong (Inactive)]

All patches has been merged.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35217/
Subject: LU-12043 llite: fix to submit complete read block with ra disabled
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7e8efb339b0958146eb294fc9d961688f5d16079