[LU-6179] Lock ahead - Request extent locks from userspace

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.11.0
• Labels: bgti patch
• Assignee: Patrick Farrell (Inactive)
• Reporter: Patrick Farrell (Inactive)

Description

At the recent developers conference, Jinshan proposed a different method of approaching the performance problems described in LU-6148.

Instead of introducing a new type of LDLM lock matching, we'd like to make it possible for user space to explicitly request LDLM locks asynchronously from the IO.

I've implemented a prototype version of the feature and will be uploading it for comments. I'll explain the state of the current version in a comment momentarily.

Issue Links

• is blocking: LU-9962 Possible file size glimpse issue
• is related to: LU-10136 sanity test_255c: Ladvise test11 failed, 255
• is related to: LUDOC-379 Document ladvise lockahead feature
• is related to: LU-6181 Fix lock contention discovery to disable extend lock growth
• is related to: LU-11618 implement ladvise rpc_size for optimized performance
• is related to: LU-6148 Strided lock proposal - Feature proposal for 2.8
• is related to: LU-7225 change ladvise wire protocol for lockahead and future usage

Comments

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38109
Subject: LU-6179 llite: remove LOCKAHEAD_OLD compatibility
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f6bc909bfda5521454631a4985648e07c63137ee

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/13564/
Subject: LU-6179 llite: Implement ladvise lockahead
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a8dcf372f430c308d3e96fb506563068d0a80c2d

[Patrick Farrell (Inactive)]

Oh. Hm. No - It's skipping some of the tests. Sorry about that, thanks for pointing it out. Some development stuff I was doing escaped in to what I pushed upstream... I'll fix that when I rebase to merge.

[Mikhail Pershin]

Patrick, the new test 255c in sanity.sh reports the following:

== sanity test 255c: suite of ladvise lockahead tests ================================================ 04:54:04 (1495688044) Starting test test10 at 1495688045
Finishing test test10 at 1495688045
Starting test test20 at 1495688045
cannot give advice: Invalid argument (22)
cannot give advice: Invalid argument (22)
cannot give advice: Invalid argument (22)
cannot give advice: Invalid argument (22)
cannot give advice: Invalid argument (22)
cannot give advice: Invalid argument (22)
Finishing test test20 at 1495688045

Is that expected or test isn't working properly?

This is from the latest test results https://testing.hpdd.intel.com/sub_tests/38e1f10a-4129-11e7-91f4-5254006e85c2

 

[Cong Xu (Inactive)]

Hi Patrick,

In the very simple test you suggested, the lock count does increase from 0 to 4010, so the lock ahead works well.
Yes, a faster OST should show more benefits.

[Patrick Farrell (Inactive)]

Huh, OK! That's a clever way to show it.

A faster OST will show much larger benefits, of course.

[Cong Xu (Inactive)]

Hi Patrick,

Thanks for the great suggestions! We conducted more tests recently and was able to demonstrate the power of Lock Ahead in test "2.3 Vary Lustre Stripe Size (Independent I/O)".

In this test, the transfer size of each process is configured to be 1MB and the stripe size grows from 256KB to 16MB. When the stripe size equals to 16MB, 16 processes write a single stripe simultaneously, leading to lock contention issues. In this test, Lock Ahead code performs 21.5% better than original code.

[Patrick Farrell (Inactive)]

Slides and paper from Cray User Group 2017 attached. They contain real performance #s on real hardware, including from real applications. Just for reference in case anyone is curious.

[Patrick Farrell (Inactive)]

Cong,

Sorry to take a bit to get back to you.

Given the #s in section 2.4, you're barely seeing the problem, and lockahead does have some overhead. I wouldn't necessarily expect it to help in that case. It would be much easier to see if you had faster OSTs - So, I'd like to request RAM backed OSTs.

It's also possible something is wrong with the library. While I think we'll need RAM backed OSTs (or at least, much faster OSTs) to see benefit, we can explore this possibility as well.

Let's take one of the very simple tests, like a 1 stripe file with 1 process per client on 2 clients. I assume you're creating the file fresh before the test, but if not, please remove it and re-create it right before the test. Then, let's look at lock count before and after running IOR (add the -k option so the file isn't deleted, otherwise the locks will be cleaned up).

Specifically, on one of the clients, cat the lock count for the OST where the file is:
cat /sys/fs/lustre/ldlm/namespaces/[OST]/lock_count

Before and after the test.

If the file is not deleted and the lock count hasn't gone up, lock ahead didn't work for some reason.

Again, I think we'll need RAM backed OSTs regardless... But this would be useful even without that.

[Cong Xu (Inactive)]

Hi Jinshan,

Thanks for the suggestions! Yes, in our second test (Section 2.2 Vary number of processes (Independent I/O)), we started from 1 process per client with 8 clients (total 8 processes) to 64 processes per client with 8 clients (total 512 processes).

Hi Patrick,

Yes, we have tried simple test as you suggested. Please have a look at the results in section 2.4: Simple Test (1 process and 2 processes accessing a single shared file on one OST).