[LU-13309] performance optimizations for brw

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: Improvement
• Priority: Minor
• Reporter: Andrew Perepechko
• Assignee: Andrew Perepechko
• Resolution: Fixed
• Labels: patch, performance

Description

A few trivial patches avoiding OSS CPU bottleneck with NVME storage will be uploaded shortly.

Issue Links

• is related to: LU-13419 Simplify osc_enter_cache_try
• is related to: LU-15532 OOM in osd_device_alloc()
• is related to: LU-13542 osd stats are initialized too late
• is related to: LU-12179 allocate continuous pages when disabl...

Comments

[Gerrit Updater]

Andrew Perepechko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37758
Subject: LU-13309 osd-ldiskfs: remove per-page object_get/put in brw
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8524ffd75fa443d02c9ce74e9e998c02d8e55b72

[Shuichi Ihara]

Andrew, what is particular workload you've seen this high CPU usages?

[Andrew Perepechko]

sihara, it's an IOR IOPS test, i.e. random read 4 KiB chunk from a 16 GiB file, single OST.

OSS configuration:
CPU based on AMD Rome, NVME storage.

Client configuration:
48 clients, 8 processes per node.

[Gerrit Updater]

Andrew Perepechko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37786
Subject: LU-13309 osd-ldiskfs: speedup osd_bufs_get/put
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7138a24a2362b8fd021e5fc000e671d309816534

[Andrew Perepechko]

One more patch, brw_stats related, will be uploaded soon.

The IOPS test shows ~590000 IOPS -> ~620000 IOPS improvement with the three patches.

Either with or without these three patches CPU is the bottleneck, 100% CPU load.

The CPU profile shows that the load is spread between various CPU consumers, the most significant of which is memset from the malloc/free paths. So further optimizations can be done by disabling memory poisoning on free (a trivial single liner) and, more interesting, by removing memset(0) from the malloc path, especially for short IO.

[Gerrit Updater]

Andrew Perepechko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37795
Subject: LU-13309 ofd: optimize the brw codepath
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8a065689c6fe38154d742f740355e58ab9f8f9d2

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37758/
Subject: LU-13309 osd-ldiskfs: remove per-page object_get/put in brw
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dc9f28a541aec49b1787a25169f56a78a4924ee4

[Gerrit Updater]

Andrew Perepechko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37915
Subject: LU-13309 osd: use per-cpu counters for brw_stats
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9f839039830b9aac5bf9038c2124a1e7a2450b0a

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37795/
Subject: LU-13309 ofd: optimize the brw codepath
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3beeb90cfa47b4753083d09760a6bd5ecaf58d76

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37786/
Subject: LU-13309 osd-ldiskfs: speedup osd_bufs_get/put
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e790df5fa38d4e8928dd28ba9f250fec4c830786

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/37915/
Subject: LU-13309 osd: use per-cpu counters for brw_stats
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 787c1884e6451ae764568ade3658e537dcc19097

[Peter Jones]

Landed for 2.15