[LU-13905] A single stream performance regression in client 4.18.0-193.14.2.el8_2 kernel

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.14.0
• Environment: RHEL8.2 (kernel kernel 4.18.0-193.14.2.el8_2) ; lustre-commit: c54b6ca (master)
• Severity: 2
• Assignee: WC Triage
• Reporter: Shuichi Ihara

Description

There is a single stream read performance regression with 4.18.0-193.14.2.el8_2 kernel. Here is test environment and a reproducer.

1 x client (1 x Gold 5218 CPU @ 2.30GHz, 96GB RAM, 1 x IB-HDR100)
CentOS8.2  (Tested kernel version: 4.18.0-147.el8.x86_64 and 4.18.0-193.14.2.el8_2.x86_64)
OFED-5.0-2.1.8.0

 

[root@ec01 ~]# lctl set_param osc.*.max_pages_per_rpc=16M osc.*.max_rpcs_in_flight=16 llite.*.max_read_ahead_mb=2048 llite.*.max_read_ahead_per_file_mb=N
[root@ec01 ~]# clush -w es400nvx1-vm[1-4],7990e3-vm[1-2],ec01 "echo 3 > /proc/sys/vm/drop_caches"
[root@ec01 ~]# /work/tools/bin/ior -r -t 1m -b 192g -e -o /es400nv/s/file -k  

At least, the behaviors with max_read_ahead_per_file_mb=64 (default) are different between two kernel versions 4.18.0-147.el8.x86_64 and 4.18.0-193.14.2.el8_2.x86_64.
Here is what I've tested on NVMe OST system.

	4.18.0-147.el8.x86_64	4.18.0-193.14.2.el8_2.x86_64
max_read_ahead_per_file_mb=64	4252(MiB/s)	2943(MiB/s)
max_read_ahead_per_file_mb=128	4186(MiB/s)	4287(MiB/s)

It was 30% slower performance with max_read_ahead_per_file_mb=64, but when it increased to 128, both performance were close.

There is another results which was tested on HDD based OSTs.

	4.18.0-147.el8.x86_64	4.18.0-193.14.2.el8_2.x86_64
max_read_ahead_per_file_mb=64	1578(MiB/s)	1326(MiB/s)
max_read_ahead_per_file_mb=128	3396(MiB/s)	2827(MiB/s)

In this case, there was still ~16% performrance regressions in 4.18.0-193.14.2.el8_2.x86_64 regardless max_read_ahead_per_file_mb=64 or 128.