[LU-11775] buffered write single client improvements

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.13.0
• Type: Improvement
• Priority: Minor
• Reporter: Dongyang Li
• Assignee: Dongyang Li
• Resolution: Fixed

Description

After landing LU-9906 the read performance improved to an acceptable level. Write performance also increased but still needs some improvements to reach the same level. This ticket tracks the work.

Issue Links

• is related to: LU-9920 Use pagevec for marking pages dirty
• is related to: LU-11942 OSC page & grant simplification/batching

Comments

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33858
Subject: LU-11775 osc: reduce lock contention in osc_unreserve_grant
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ffcc069b2b654f6437b090b8ff8c67b5d1ac2ea6

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33859
Subject: LU-11775 osc: reduce atomic ops in osc_enter_cache_try
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 95d0af71efca35ea610bbddec2025100b052cdf5

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33860
Subject: LU-11775 osc: check if opg is in lru list without locking
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4225477391bd60784151a6102f1fc989e11f7a92

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33861
Subject: LU-11775 obdclass: protect imp_sec using rwlock_t
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 87b1a916e6955c10943d6ee6b9b4d9af393c93cc

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33862
Subject: LU-11775 osc: check imp_invalid without imp_lock
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: db8dda7213403e2b2caedf33b55e4fa5f2e62db6

[Dongyang Li]

As we can see LU-9920 can reduce the overhead in set_page_dirty/write_commit_callback.

There is still __ _raw_spin_lock from osc_page_cache_add which I can not figure out the exact location.

mpirun -np 32 ior -w -t 16m -b 16g -F -e -vv -o /cache1/file -k

master:

Max Write: 15237.92 MiB/sec (15978.12 MB/sec)

master + LU-11775:

Max Write: 17045.11 MiB/sec (17873.09 MB/sec)

master + LU-11775 + LU-9920:

Max Write: 17476.77 MiB/sec (18325.72 MB/sec)

[Patrick Farrell (Inactive)]

Li,

Looking at your graphics, I think maybe osc_lru_use has been inlined in to osc_page_cache_add.  I haven't dug in to "LU-11775 osc: check if opg is in lru list without locking" yet, but I don't think it would stop us from hitting that lock entirely.

If you'd like to avoid inlining, the easiest way I've found is calling that function via a function pointer.  Then you could see it easily in your traces again.  (There are various "don't inline" directives, but I've never been able to get them to work when building Lustre.)

[Dongyang Li]

Patrick, thanks for the nice tip!

Yes I've been using noinline directive but apparently it did not work.

I will give it a shot when I can get to the environment, and provide an update on LU-9920 as well.

Cheers

DY

[Andreas Dilger]

master:
Max Write: 15237.92 MiB/sec (15978.12 MB/sec)

master + LU-11775:
Max Write: 17045.11 MiB/sec (17873.09 MB/sec)

master + LU-11775 + LU-9920:
Max Write: 17476.77 MiB/sec (18325.72 MB/sec)

This information should go into the commit comments of the various patches, like https://review.whamcloud.com/28667

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33859/
Subject: LU-11775 osc: reduce atomic ops in osc_enter_cache_try
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8b364fbd6bd9e0088440e6d6837861a641b923a0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33860/
Subject: LU-11775 osc: check if opg is in lru list without locking
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b3af0798682b24d0f39424a7db12c9710a0e64b4

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33858/
Subject: LU-11775 osc: reduce lock contention in osc_unreserve_grant
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8a1ae45a3e4f828346c890bf4ccacbd68ea6752c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33861/
Subject: LU-11775 obdclass: protect imp_sec using rwlock_t
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8ed361345154138473707855b2030bad371e96dd

[Andreas Dilger]

Dongyang, is there still work to be done for this ticket? It looks like the patches are included in 2.13.

[Dongyang Li]

No more patches and I think this ticket can be closed now