[LU-8515] OSC: Send RPCs with full extents

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.10.0
• Type: Bug
• Priority: Major
• Reporter: Patrick Farrell (Inactive)
• Assignee: Patrick Farrell (Inactive)
• Resolution: Fixed
• Severity: 3

Description

In Lustre 2.7 and newer, single node multi-process single-shared-file write performance is significantly slower than in Lustre 2.5. This is due to a problem in deciding when to make an RPC. (IE, the decisions made in osc_makes_rpc)

Currently, Lustre decides to send an RPC under a number of
conditions (such as memory pressure or lock cancellcation);
one of the conditions it looks for is "enough dirty pages
to fill an RPC". This worked fine when only one process
could be dirtying pages at a time, but in newer Lustre
versions, more than one process can write to the same
file (and the same osc object) at once.

In this case, the "count dirty pages method" will see there
are enough dirty pages to fill an RPC, but since the dirty
pages are being created by multiple writers, they are not
contiguous and will not fit in to one RPC. This resulted in
many RPCs of less than full size being sent, despite a
good I/O pattern. (Earlier versions of Lustre usually
send only full RPCs when presented with this pattern.)

Instead, we remove this check and add extents to a special
full extent list when they reach max pages per RPC, then
send from that list. (This is similar to high priority
and urgent extents.)

With a good I/O pattern, like usually used in benchmarking,
it should be possible to send only full size RPCs. This
patch achieves that without degrading performance in other
cases.

In IOR tests with multiple writers to a single file,
this patch improves performance by several times, and
returns performance to equal levels (single striped files)
or much greater levels (very high speed OSTs, files
with many stripes) vs earlier versions.

Here's some specific data:
On this machine and storage system, the best bandwidth we can get to a single stripe from one node is about 330 MB/s. This occurs with one writer. All tests are run on a newly created, singly striped file, except where a higher stripe count is specified.

IOR: aprun -n 1 $(IOR) -w -t 4m -b 16g -C -e -E -k -u -v
(1 thread, 4 MiB transfer size, 16GB per thread.)

Unmodified:
write 334.12 334.12 334.12 0.00 83.53 83.53
write 329.34 329.34 329.34 0.00 82.33 82.33
write 329.37 329.37 329.37 0.00 82.34 82.34

Modified (full extent):
write 329.47 329.47 329.47 0.00 82.37 82.37
write 339.33 339.33 339.33 0.00 84.83 84.83
write 323.18 323.18 323.18 0.00 80.80 80.80

Here's an example of the improvement available. We're using 8 threads and 1 GB of data per thread. (Results are similar with a larger amount of data per thread.)
IOR: aprun -n 8 $(IOR) -w -t 4m -b 1g -C -e -E -k -u -v
Unmodified:
write 87.24 87.24 87.24 0.00 21.81 21.81
write 89.26 89.26 89.26 0.00 22.31 22.31
write 90.45 90.45 90.45 0.00 22.61 22.61

Modified:
write 345.72 345.72 345.72 0.00 86.43 86.43
write 334.14 334.14 334.14 0.00 83.53 83.53
write 351.03 351.03 351.03 0.00 87.76 87.76

Note the above is actually a shade higher than the single thread performance, despite being at essentially the limit for the target (from this node, with these settings).

2 stripes:

1 thread, unmodified:
write 614.48 614.48 614.48 0.00 153.62 153.62
write 626.98 626.98 626.98 0.00 156.75 156.75
write 610.14 610.14 610.14 0.00 152.53 152.53

1 thread, modified:
write 627.86 627.86 627.86 0.00 156.97 156.97
write 625.68 625.68 625.68 0.00 156.42 156.42
write 625.47 625.47 625.47 0.00 156.37 156.37

8 threads, unmodified:
write 172.24 172.24 172.24 0.00 43.06 43.06
write 180.02 180.02 180.02 0.00 45.01 45.01
write 186.17 186.17 186.17 0.00 46.54 46.54

8 threads, modified:
write 614.53 614.53 614.53 0.00 153.63 153.63
write 604.05 604.05 604.05 0.00 151.01 151.01
write 616.77 616.77 616.77 0.00 154.19 154.19

8 stripes:
Note - These tests were run with 4 or 8 GB of data per thread, otherwise they completed too quickly for me to be comfortable (though the numbers were similar). Performance numbers were the same across all total amounts of data tested. Numbers given below are representative - I repeated each test several times, but didn't want to put in that much data.

1 thread, unmodified:
write 1270.16 1270.16 1270.16 0.00 317.54 317.54

1 thread, modified:
write 1256.26 1256.26 1256.26 0.00 314.06 314.06

8 threads, unmodified:
write 712.33 712.33 712.33 0.00 178.08 178.08

8 threads, modified:
write 1949.85 1949.85 1949.85 0.00 487.46 487.46

16 stripes:

8 threads, unmodified:
write 1461.83 1461.83 1461.83 0.00 365.46 365.46

8 threads, modified:
write 3082.42 3082.42 3082.42 0.00 770.61 770.61

Comments

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22012
Subject: LU-8515 osc: Send RPCs when extents are full
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eee670db16607a21331c265a1cb041b8654cc586

[Jinshan Xiong (Inactive)]

Were these tests running with 4MB transfer size, and the max_pages_per_rpc is set to the default value that is 256?

[Patrick Farrell (Inactive)]

Yes, they were all done with 4 MiB transfer size and 1 MiB RPCs (So, 256 max_pages_per_rpc). I can't increase the RPC size on this system, but I can reduce the transfer size to 1 MiB, so they're matched. (Note that stripe size is 1 MiB.)

Here's (a few of) those tests repeated with 1 MiB transfer size. If you have specific tests you'd like run, let me know and I can try to get time later. I can also get some rpc_stats data for a few tests if needed.

For the modified version, the results are the same. For the unmodified version, speed is up a bit across the board, but still a lot slower than the modified version. This is expected, since with 4 MiB transfers, every write touches multiple stripes (all but guaranteeing multiple active extents in each osc object at the same time, which causes us to send small RPCs). With 1 MiB transfers, this doesn't happen all the time - But it does happen some of the time. (When writers, stripe sizes, and stripe counts match up, like in the 8 stripe case, I'm not clear on why we still we have problems. I would except the writers not to interfere in that case - But the results below show they seem to. Probably needs separate investigation.)

Singly striped:

1 thread, 1 MiB transfer size. (These results should be the same. They tend to be, within the margin of error.)

Unmodified:
write 337.20 337.20 337.20 0.00 337.20 337.20 337.20 0.00 24.29393 1 1 1 0 1 1 0 0 1 0 1048576 8589934592 -1 POSIX EXCEL

Modified:
write 360.45 360.45 360.45 0.00 360.45 360.45 360.45 0.00 22.72692 1 1 1 0 1 1 0 0 1 0 1048576 8589934592 -1 POSIX EXCEL

Here's where we see the difference.

8 threads, 1 MiB transfer size:
Unmodified:
write 103.66 103.66 103.66 0.00 103.66 103.66 103.66 0.00 79.02997 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

Modified:
write 342.51 342.51 342.51 0.00 342.51 342.51 342.51 0.00 23.91763 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

8 stripes:

8 threads, 1 MiB transfer size:
Unmodified:
write 869.89 869.89 869.89 0.00 869.89 869.89 869.89 0.00 9.41728 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

Modified:
write 1882.86 1882.86 1882.86 0.00 1882.86 1882.86 1882.86 0.00 4.35084 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

16 stripes:

8 threads, 1 MiB transfer size:
Unmodified:
write 2062.42 2062.42 2062.42 0.00 2062.42 2062.42 2062.42 0.00 31.77620 8 8 1 0 1 1 0 0 1 0 1048576 68719476736 -1 POSIX EXCEL

Modified:
write 3092.02 3092.02 3092.02 0.00 3092.02 3092.02 3092.02 0.00 21.19523 8 8 1 0 1 1 0 0 1 0 1048576 68719476736 -1 POSIX EXCEL

[Patrick Farrell (Inactive)]

By the way, from LU-1669:
https://jira.hpdd.intel.com/browse/LU-1669?focusedCommentId=57794&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-57794

SIDE NOTE:

As an aside, I ran into what appeared to be poor RPC formation when writing to a single shared file from many different threads, and simultaneously hitting the per OSC dirty page limit. During "Test 5" (and to a lesser extent, "Test 2" as well) I began to see many non 1M RPCs being sent to the OSS nodes, whereas with the other tests, nearly all of the RPCs were 1M in size. This affect got worse as the number of tasks increased.

What I think was happening is this

• As the client pushes data fast enough to the server, it bumps up
  against the per OSC dirty limit, thus RPCs are forcefully flushed out

• As this is happening, threads are continuously trying to write data
  to there specific region of the file. Some tasks are able to fill a
  full 1M buffer before the dirty limit forces a flush, but some tasks
  are not.

• Buffers to non-contiguous regions of a file are not joined together,
  so the smaller non-1M buffers are forced out in non-optimal small
  RPCs.

I believe this affect was only apparent in Test 2 and Test 5, because the other tests just weren't able to push data to the server fast enough to bump up against the dirty limit.

It would be nice if the RPC formation engine would keep these small buffers around, waiting for them to reach a full 1M, before flushing them out to the server. This is especially harmful on ZFS backends because it can force read-modify-write operations, as opposed to only performing writes when the RPC is properly aligned at 1M.

{/quote}

He was partly right. It turns out the main reason for non-optimal RPC sizes was actually this choice about when to send in osc_makes_rpc.

[Jinshan Xiong (Inactive)]

The client chose to send RPC earlier is because there is no more grant, or dirty pages has reached its limit therefore it can't cache more dirty data.

The problem with your patch is that it may cause livelock - server is in short of space and there is no chance for this client to make full RPC, and then individual threads will hold their own partial extents, and then nobody can move forward.

[Patrick Farrell (Inactive)]

Jinshan,

Hmmm.

I don't think this change affects the behavior of the client in the case where we're out of grant/dirty pages has reached its limit. Those cases are handled separately in osc_makes_rpc. In that case, we send an RPC due to cl_cache_waiters.

The check I replaced is this one:
if (atomic_read(&osc->oo_nr_writes) >=
cli->cl_max_pages_per_rpc)

Which is just to send an RPC when we have enough dirty pages for one. We already decide to send an RPC when there are cache waiters.

Are you saying there is a case where we must write out some data, but we do not generate hp exts, urgent exts, or have cache waiters? I think that's what would be required to get a livelock. If so, then the existing check won't make us safe either. It's a per-object check that won't fire unless there are >= cl_max_pages_per_rpc in that object. So with the current code, a large number of objects could cause the livelock you describe, if all of them had a small amount of data written to them. (Since this check wouldn't catch that.)

[Jinshan Xiong (Inactive)]

We already decide to send an RPC when there are cache waiters.

The function osc_makes_rpc() just tells the I/O engine it could be possible to issue an BRW RPC, and then the I/O engine scans the OSC page cache to try to compose an RPC. It doesn't guarantee that it can make one.

With this in mind, I think we should keep the code:

if (atomic_read(&osc->oo_nr_writes) >= cli->cl_max_pages_per_rpc)

but it's a really good idea to have a full extent list and compose RPCs from that list first.

Anyway, I think your observation is valuable, and your patch makes a lot of sense to me, just need to tweak a little bit.

[Jinshan Xiong (Inactive)]

btw, is it possible to write a few test cases based on your findings so that we won't break this in the future?

[Patrick Farrell (Inactive)]

About test cases:
Yes, I think it's probably possible to do that. I'll have to think about it - I think it would be something like "run an IOR job with a good I/O pattern and check the rpc sizes". Make sure, say, 90 or 95% of RPCs are maximum size. (I get 99%-100% in my testing.)

[Patrick Farrell (Inactive)]

The function osc_makes_rpc() just tells the I/O engine it could be possible to issue an BRW RPC, and then the I/O engine scans the OSC page cache to try to compose an RPC. It doesn't guarantee that it can make one.

Right.

With this in mind, I think we should keep the code:

if (atomic_read(&osc->oo_nr_writes) >= cli->cl_max_pages_per_rpc)

but it's a really good idea to have a full extent list and compose RPCs from that list first.

I don't think we can do that. I can try adding it back in and running some benchmarks (let me know if you want me to do that), but we call osc_makes_rpc for every page we dirty. So we'll try to make an RPC, and (in get_write_extents) if there are no full extents, we'll still send some data. So we'll send our extents before they get to full size.

[Jinshan Xiong (Inactive)]

but we call osc_makes_rpc for every page we dirty.

It's a bug if this happens - we should only call this whenever it's possible to make an RPC, for example, releasing a osc_extent, brw_interpret(), or other urgent cases.

Yes, please do the tests with my suggestions, it will help me understand the code better.

[Patrick Farrell (Inactive)]

Ah, right, sorry. I misread the code (there are a lot of ways to call osc_makes_rpc...). You're right about when we call it.

So it probably is safe (in terms of performance) to add back that check, as long as we track full extents and try to send them first in get_write_extents.

I'll try it - it would just replace checking list_empty in osc_makes_rpc. If it works, then it doesn't really matter which check we have there, the current one or checking the full_ext list directly.

[Patrick Farrell (Inactive)]

With this check, I think we're probably getting a few more non-optimally sized RPCs.

This is because at the end of brw_interpret, osc_io_unplug is called, and that results in calling osc_check_rpcs. So each time a ptlrpcd thread completes a brw write, it calls osc_check_rpcs, which calls osc_makes_rpc.

So that's one case we call osc_makes_rpc when we don't know if anything is ready for us. So I think with the oo_nr_writes check, we will sometimes send small RPCs with incomplete extents (that we're still writing to).

Also, I think the oo_nr_writes >= cl_max_pages_per_rpc check is specifically trying to send when we expect to send out a full size RPC. So I think it's better to just check for 'full extents' - I think it describes what we're really doing better.

But if we do keep the oo_nr_writes check, I think it will work fine.

[Patrick Farrell (Inactive)]

Sorry about deleting that comment. It turns out I ran the tests above in FPP node, not SSF mode. Ugh.

Rerunning the tests, I see a significant difference in favor of not using the oo_nr_writes check. Until we hit the bandwidth limit of the node (16 stripes), !list_empty(full_ext) is significantly faster.

My reason for why is still the call to osc_makes_rpc we get every time we call brw_interpret. I think that's generating a lot of non-optimal RPCs.

Here's some data, using 1 MiB transfer sizes (4 MiB was very similar):
1 stripe, 8 process:
oo_nr_writes
write 90.97 90.97 90.97 0.00 90.97 90.97 90.97 0.00 45.02345 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 90.97 90.97 90.97 0.00 90.97 90.97 90.97 0.00 45.02626 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 90.29 90.29 90.29 0.00 90.29 90.29 90.29 0.00 45.36592 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
^-- This test gives about 70 MB/s with no changes at all.

!list_empty(full_ext):
write 346.29 346.29 346.29 0.00 346.29 346.29 346.29 0.00 11.82837 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 350.43 350.43 350.43 0.00 350.43 350.43 350.43 0.00 11.68849 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 350.00 350.00 350.00 0.00 350.00 350.00 350.00 0.00 11.70296 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL

8 stripes, 8 processes:
oo_nr_writes
write 691.72 691.72 691.72 0.00 691.72 691.72 691.72 0.00 5.92146 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 642.44 642.44 642.44 0.00 642.44 642.44 642.44 0.00 6.37567 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 602.00 602.00 602.00 0.00 602.00 602.00 602.00 0.00 6.80393 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL

!list_empty(full_ext):
write 1844.41 1844.41 1844.41 0.00 1844.41 1844.41 1844.41 0.00 2.22076 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 1939.05 1939.05 1939.05 0.00 1939.05 1939.05 1939.05 0.00 2.11238 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 1866.54 1866.54 1866.54 0.00 1866.54 1866.54 1866.54 0.00 2.19443 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL

16 stripes, 8 processes:
oo_nr_writes:
write 2619.29 2619.29 2619.29 0.00 2619.29 2619.29 2619.29 0.00 1.56378 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 3091.95 3091.95 3091.95 0.00 3091.95 3091.95 3091.95 0.00 1.32473 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 3039.58 3039.58 3039.58 0.00 3039.58 3039.58 3039.58 0.00 1.34756 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 3171.61 3171.61 3171.61 0.00 3171.61 3171.61 3171.61 0.00 2.58292 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3154.39 3154.39 3154.39 0.00 3154.39 3154.39 3154.39 0.00 2.59701 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3138.18 3138.18 3138.18 0.00 3138.18 3138.18 3138.18 0.00 2.61043 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

!list_empty(full_ext):
write 2896.65 2896.65 2896.65 0.00 2896.65 2896.65 2896.65 0.00 1.41405 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 2974.45 2974.45 2974.45 0.00 2974.45 2974.45 2974.45 0.00 1.37706 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 3064.65 3064.65 3064.65 0.00 3064.65 3064.65 3064.65 0.00 1.33653 8 8 1 0 1 1 0 0 1 536870912 1048576 4294967296 -1 POSIX EXCEL
write 2814.37 2814.37 2814.37 0.00 2814.37 2814.37 2814.37 0.00 2.91078 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3078.64 3078.64 3078.64 0.00 3078.64 3078.64 3078.64 0.00 2.66092 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3211.55 3211.55 3211.55 0.00 3211.55 3211.55 3211.55 0.00 2.55079 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

16 stripes, 16 processes:
oo_nr_writes:
write 3145.50 3145.50 3145.50 0.00 3145.50 3145.50 3145.50 0.00 2.60436 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3137.90 3137.90 3137.90 0.00 3137.90 3137.90 3137.90 0.00 2.61066 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3233.13 3233.13 3233.13 0.00 3233.13 3233.13 3233.13 0.00 2.53377 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

!list_empty(full_ext):
write 2940.08 2940.08 2940.08 0.00 2940.08 2940.08 2940.08 0.00 2.78631 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3149.34 3149.34 3149.34 0.00 3149.34 3149.34 3149.34 0.00 2.60118 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL
write 3132.14 3132.14 3132.14 0.00 3132.14 3132.14 3132.14 0.00 2.61546 8 8 1 0 1 1 0 0 1 1073741824 1048576 8589934592 -1 POSIX EXCEL

[Andreas Dilger]

Patrick, do you have any before/after testing done with clients doing sub-RPC write size (e.g. interleaved 256KB per client so that they don't merge into a single full RPC)? One problem that we had in the past was that sub-sized dirty pages filling up memory without being flushed in a timely manner, so I just want to make sure that the new code that is deferring RPCs until they are full is not over-zealous in delaying dirty data that don't make up a full RPC.

[Patrick Farrell (Inactive)]

No, but I will try to get some. Should be quick once machines are available today. (I am pretty confident this should be OK. The new code only waits longer than the old code in the case of multiple writers per stripe. I don't know what ensures pages go out in a timely manner, but I don't think I've modified it.)

Can you clarify what you mean by interleaved? Other than by doing direct and buffered I/O, I can't think of how to prevent extents in that situation being packaged in to one RPC. I suppose I/O from two different clients could sort of do that, but I think we'd actually get data written out via ldlm lock cancellation. (I could use group locks to avoid that.)

[Andreas Dilger]

By interleaved writes I mean having client A write [0,256KB), client B write [256KB,512KB), ... so that the writes cannot be merged on the client into a single 1MB RPC. That doesn't test the issue I'm wondering about if these writes are just being done by different threads on the same client.

[Patrick Farrell (Inactive)]

OK. My concern with that is that due to ldlm lock exchange (think lock ahead), those bytes will be written out by lock cancellation, so they won't have a chance to hang around anyway.

I'll see about running such a job and checking the results.

[Jinshan Xiong (Inactive)]

One problem that we had in the past was that sub-sized dirty pages filling up memory without being flushed in a timely manner

In that case, write back daemon should be started to writeback pages, and those pages should appear in urgent list that shouldn't be affected by this patch.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/22012/
Subject: LU-8515 osc: Send RPCs when extents are full
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ecb6712a19fa836ecdba41ccda80de0a10b1336a

[Peter Jones]

Landed for 2.10