Details
-
Bug
-
Resolution: Fixed
-
Minor
-
Lustre 2.15.2
-
None
-
Lustre-2.15.2, Rokeylinux 8.6 (4.18.0-372.32.1.el8_6.x86_64), OFED-5.4-3.6.8.1
-
3
-
9223372036854775807
Description
a client performance regression was found in 2.15.2-RC1 (commit:e21498bcaa).
Tested workload is single client and SSF(single shared file) from 16 processes.
# mpirun -np 16 ior -a POSIX -i 1 -d 10 -w -r -b 16g -t 1m -C -Q 17 -e -vv -o //exafs/d0/d1/d2/ost_stripe/file
lustre-2.15.1
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 2489.25 2489.28 0.006428 16777216 1024.00 0.000936 105.31 0.000238 105.31 0 read 4176 4176 0.003803 16777216 1024.00 0.001695 62.77 3.92 62.77 0 write 2423.58 2423.60 0.006452 16777216 1024.00 0.000586 108.16 2.45 108.16 1 read 4197 4197 0.003652 16777216 1024.00 0.001982 62.46 3.98 62.46 1 write 2502.32 2502.34 0.006375 16777216 1024.00 0.000404 104.76 0.305282 104.76 2 read 4211 4211 0.003683 16777216 1024.00 0.001679 62.25 3.99 62.25 2 Max Write: 2502.32 MiB/sec (2623.88 MB/sec) Max Read: 4211.19 MiB/sec (4415.75 MB/sec)
lustre-2.15.2-RC1
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 2103.65 2103.68 0.007142 16777216 1024.00 0.001769 124.61 7.60 124.61 0 read 4204 4204 0.003159 16777216 1024.00 0.001461 62.35 10.59 62.35 0 write 2169.58 2169.69 0.006903 16777216 1024.00 0.000912 120.82 7.72 120.83 1 read 4282 4282 0.003722 16777216 1024.00 0.137671 61.22 2.78 61.22 1 write 2133.24 2133.25 0.007500 16777216 1024.00 0.000380 122.88 3.60 122.89 2 read 4088 4088 0.003689 16777216 1024.00 0.001053 64.13 3.68 64.13 2 Max Write: 2169.58 MiB/sec (2274.97 MB/sec) Max Read: 4282.19 MiB/sec (4490.20 MB/sec)
it is ~14% performance regression in 2.15.2-RC1 compared to lustre-2.15.1.
After investigations and 'git bisect' tells us "commit: [6d4559f6b948a93aaf5e94c4eb47cd9ebcf7ba95] LU-15959 kernel: new kernel [SLES15 SP4 5.14.21-150400.24.18.1]" caused this performance regression.
Here is another test result after revered patch "LU-15959 kernel: new kernel [SLES15 SP4 5.14.21-150400.24.18.1]" from lustre-2.15.2-RC1 and it confirmed the performance was back to same level of 2.15.1.
lustre-2.15.2-RC1 + reverted commit:6d4559f6b9 (LU-15959 kernel: new kernel [SLES15 SP4 5.14.21-150400.24.18.1])
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 2497.41 2497.44 0.006407 16777216 1024.00 0.001115 104.97 0.000791 104.97 0 read 4217 4217 0.003773 16777216 1024.00 0.001680 62.16 3.37 62.16 0 write 2471.13 2471.14 0.006475 16777216 1024.00 0.000375 106.08 0.000292 106.08 1 read 4083 4083 0.003765 16777216 1024.00 0.001659 64.20 3.23 64.20 1 write 2457.91 2457.92 0.006509 16777216 1024.00 0.000412 106.65 0.010367 106.65 2 read 4163 4163 0.003771 16777216 1024.00 0.001909 62.97 6.35 62.97 2 Max Write: 2497.41 MiB/sec (2618.72 MB/sec) Max Read: 4217.39 MiB/sec (4422.25 MB/sec)
Attachments
Issue Links
- is related to
-
LU-15959 support for SLES 15 SP4
-
- Resolved
-
Activity
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49520/
Subject: LU-16433 llite: check vvp_account_page_dirtied
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 1c6e03a53cb374c10cf2d9e5a22fdb304f81e8bf
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49512/
Subject: LU-16433 llite: check vvp_account_page_dirtied
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 61c4c2b5e5d7d919149921d913322586ba5ebcab
"Xing Huang <hxing@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49520
Subject: LU-16433 llite: check vvp_account_page_dirtied
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: b95cf135117cd24ef5403aa111ae82fd14215efb
confirmed that patch https://review.whamcloud.com/c/fs/lustre-release/+/49512 solved problem and performance was back.
lustre-2.15.2-RC1 + patch https://review.whamcloud.com/c/fs/lustre-release/+/49512
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- write 2440.73 2440.76 0.006555 16777216 1024.00 0.001139 107.40 0.000249 107.40 0 read 4027 4027 0.003897 16777216 1024.00 0.001635 65.09 3.60 65.09 0 write 2427.14 2427.15 0.006584 16777216 1024.00 0.000384 108.00 0.126996 108.01 1 read 4132 4132 0.003715 16777216 1024.00 0.001663 63.44 5.11 63.44 1 write 2421.75 2421.76 0.006581 16777216 1024.00 0.000384 108.25 1.39 108.25 2 read 4082 4082 0.003875 16777216 1024.00 0.001668 64.22 3.72 64.22 2 Max Write: 2440.73 MiB/sec (2559.29 MB/sec) Max Read: 4132.11 MiB/sec (4332.83 MB/sec)
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49512
Subject: LU-16433 llite: define and check vvp_account_page_dirtied
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 11b721714311ce9f11a596eaa13c368d27096d96
In patch https://review.whamcloud.com/47924 ("LU-15959 kernel: new kernel [SLES15 SP4 5.14.21-150400.24.18.1]"), the following changes are related:
-#ifndef HAVE_ACCOUNT_PAGE_DIRTIED_EXPORT +#if !defined(HAVE_ACCOUNT_PAGE_DIRTIED_EXPORT) || \ +defined(HAVE_KALLSYMS_LOOKUP_NAME) extern unsigned int (*vvp_account_page_dirtied)(struct page *page, struct address_space *mapping); #endif
/* kernels without HAVE_KALLSYMS_LOOKUP_NAME also don't have account_dirty_page * exported, and if we can't access that symbol, we can't do page dirtying in * batch (taking the xarray lock only once) so we just fall back to a looped * call to __set_page_dirty_nobuffers */ #ifndef HAVE_KALLSYMS_LOOKUP_NAME for (i = 0; i < count; i++) __set_page_dirty_nobuffers(pvec->pages[i]); #else + /* + * In kernel 5.14.21, kallsyms_lookup_name is defined but + * account_page_dirtied is not exported. + */ + if (!vvp_account_page_dirtied) { + for (i = 0; i < count; i++) + __set_page_dirty_nobuffers(pvec->pages[i]); + goto end; + } +
In Rocky Linux 8.6 kernel 4.18.0-372.32.1.el8_6.x86_64, both account_page_dirtied and kallsyms_lookup_name are exported. So, I need to change the check of vvp_account_page_dirtied to HAVE_ACCOUNT_PAGE_DIRTIED_EXPORT. This can resolved the client performance regression issue on Rocky Linux 8.6.
However, for SLES15 SP4 client, I'm not sure how to resolve the issue since account_page_dirtied is not exported and we have to use __set_page_dirty_nobuffers.
I'm working on a patch to fix the issue on Rocky Linux 8.6.
Landed for 2.16