|
I was just thinking the best way to pursue this is to probably get some ftrace/systemtap profiling data for the cl_io_loop() function. This would allow us to easily identify any call paths which are blocked for a long time with this write mutex held. At a minimum those places should be made asynchronous.
|
|
Jinshan
Please can you comment on this one?
Thanks
Peter
|
|
From the backtrace, the process was blocking at osc_enter_cache() for more grants or budget of dirty_page. lli_write_mutex is used to keep write atomicity for lustre.
There may be several causes to limit the speed of generating dirty pages:
1. llite_write_mutex (and inode mutex) serializes the write processes, this seems unavoidable if you want posix compatibility;
2. per OSC maximum dirty pages, this can be adjusted by lproc of osc.max_dirty_mb;
3. system maximum dirty pages, this can be set by max_dirty_mb under /proc/sys/lustre
Can you please try to adjust the above parameters to see if it helps? It'll be really bad if it turns out that lli_write_mutex or inode mutex limits the write speed because it would be hard to improve it.
|
|
> 1. llite_write_mutex (and inode mutex) serializes the write processes, this seems unavoidable if you want posix compatibility;
Not exactly, a mutex is just the simplest way to achieve it. There are other methods, for example doing range locking would be one way to allow far greater parallelism.
> 2. per OSC maximum dirty pages, this can be adjusted by lproc of osc.max_dirty_mb;
> 3. system maximum dirty pages, this can be set by max_dirty_mb under /proc/sys/lustre
Yes I had the same thought when running these tests originally. I checked both the dirty limits and the available grant and neither were being regularly exhausted. We even increased the stripe width of the shared file to rule this out as a significant cause, it didn't help. So while the above stack does show us blocked in osc_enter_cache(). I suspect this is more the cumulative amount of time spent with the write mutex held in cl_io_loop().
Speeding up cl_io_loop() will of course help, but only up to a point. Since we're serialized by a single mutex shared file performance will be limited to:
|
|
The difficult thing is we have to talk to kernel for VM support so it's hard to skip inode mutex. I would say if we removed lli_write_sem, inode mutex would definitely be the hotest lock as what've seen for lli_write_sem.
To apply range lock, we can't call vfs interfaces any more, so this will need a lot of work as lustre will have to talk to VM itself. However, this is not a totally bad thing because we can do better IO optimization.
|
|
I can easily believe that if lli_write_sem was converted in to a range lock then we'd see the inode mutex become the hottest lock. However, that lock is taken much less broadly so I'd expect we'd still see significant performance improvements. Perhaps enough to improve things to an acceptable level. We'd have to run the benchmarks to see.
However, I'm not quite sure I follow why using a range lock would force us to change the VFS interface we use.
|
|
> However, I'm not quite sure I follow why using a range lock would force us to change the VFS interface we use.
Now I think I understand what your getting at.
I took the opportunity to drop the lli_write_sem entirely just to see what that would do to the single shared file write performance. It turns out it barely helps. The contention just shifts as you expected to the i_mutex in generic_file_aio_write(). Presumably this is what you meant about us no longer being able use certain VFS interfaces.
|
|
I checked kernel source just now and I think we can do a quick test by calling __generic_file_aio_write() instead of generic_file_aio_write() to skip inode mutex. In this case, we can work out a range lock patch in lustre really easy.
|
|
Good thought, I hadn't noticed that __generic_file_aio_write() was also exported. I made this quick change and ran the test to see what this would buy us, the results are interesting.
The single shared file write performance does increase from ~550MB/s to over 1,300MB/s (over 2x) as observed by the client. But after about 30 seconds of run time it collapses back down to ~550MB/s. Observing the servers during this process shows that they never exceed ~550MB/s. My assumption was this was simply getting cached on the client but /proc/meminfo doesn't show a large build up of dirty pages so there's a mystery there. The servers are easily capable of handling the increase load since the shared file is striped 8 wide.
More profiling is clearly needed. However, a quick dump of the client stack traces show the current contention point is now ll_inode_size_lock(). There are a few callers blocked in osc_enter_cache() still which probably should also be explained since it appears we're rarely out of grant or dirty more than a meta byte or so per osc.
|
|
What's the status of this performance issue at this point (for 2.4.0)?
|
|
As far as I know, there hasn't been any real work done to improve this issue yet. I'm hoping to get a chance to work on this soon since it is a priority for us at LLNL, but that wont definitely won't happen in time for 2.4.0. I can't comment on where this falls on Intel's list of priority's though.
|
|
Before I lose track of this, I wanted to report on some single shared file testing I did last week.
Specifically, I ran 5 different tests. Two of which were using a client built from an unmodified version of master to serve as a baseline, and then 3 other tests were run with slight modifications to the lustre client. The point of the testing was to try and scope out what changes might be needed for the single shared file performance to match that of the file per process performance, using a single client.
For each test described above, 7 FIO iterations were run, with each iteration varying the number of tasks. To keep the total amount of data transfered over the network constant as the number of tasks increased, the amount of data each task wrote decreased proportionally.
My take away from this testing was – In order to improve the single shared file write performance of a single lustre client, we need to address these 3 areas of contention:
Below are the results of each iteration of FIO for each Test. It shows the number of tasks used, amount of data each task wrote, and the total aggregate bandwidth (as reported by FIO) for each iteration:
SIDE NOTE:
As an aside, I ran into what appeared to be poor RPC formation when writing to a single shared file from many different threads, and simultaneously hitting the per OSC dirty page limit. During "Test 5" (and to a lesser extent, "Test 2" as well) I began to see many non 1M RPCs being sent to the OSS nodes, whereas with the other tests, nearly all of the RPCs were 1M in size. This affect got worse as the number of tasks increased.
What I think was happening is this
I believe this affect was only apparent in Test 2 and Test 5, because the other tests just weren't able to push data to the server fast enough to bump up against the dirty limit.
It would be nice if the RPC formation engine would keep these small buffers around, waiting for them to reach a full 1M, before flushing them out to the server. This is especially harmful on ZFS backends because it can force read-modify-write operations, as opposed to only performing writes when the RPC is properly aligned at 1M.
|
|
Prakash, thank you very much for this great work. We can clearly see that without those locks the single shared file performance is even better than file-per-process performance.
The small write RPC would be due to dirty limit or lack of grant. This can be simply verified by increasing dirty limit to a bigger number. Actually after IO engine work is done, the OSC tend to put all pages from an IO into one extent so that they can be written out in one RPC, however, the extent has to be split if there is no enough grant or reach dirty limit.
As you know, the reason we need inode mutex and write_sem is to make lustre client be posix compliant; but maybe we can investigate the necessity of lli_size_sem.
|
We can clearly see that without those locks the single shared file performance is even better than file-per-process performance.
Keep in mind the small sample size of each iteration. Although the raw numbers show Test 5 greater than Test 2, I wouldn't put much emphasis on that fact with only a single sample for each iteration. I'm more focused on the trend in better performance up to the point where it resembles that of the file per process test.
The small write RPC would be due to dirty limit or lack of grant.
Agreed, and it sounds like you are aware of this. I had not been exposed to this affect, so I was surprised at first. In my case, even when increasing the dirty limit to 1G per OSC I still hit the dirty limit. This had the side effect of causing server performance to be much much worse because read-modify-writes were occurring when rewriting files. Simply using brand new files removed the server side performance artifact, but the small RPC sizes maintained.
As you know, the reason we need inode mutex and write_sem is to make lustre client be posix compliant; but maybe we can investigate the necessity of lli_size_sem.
Right. Without looking into the details, my initial thoughts are this:
1. lli_write_sem - This can be converted into a range lock. This will allow multiple threads to simultaneously write to a single file, as long as they are writing to unique offset ranges in the file. All while maintaining posix compatibility. If multiple threads overlap their writes, then we degerate to the point we're at now (but there's not much we can do about that case). I'm hoping I can make use of the kernel fcntrl/flock functions and implementation, rather than rolling our own custom range locking implementation.
2. inode mutex - I'm unsure if we can simply drop this as I did in my test. I need to look at the kernel sources more closely and determine exactly why they take the lock, and see if there is something clever we can do to get around this. I'm curious how local filesystems perform running the same FIO test, because I would assume they'll run into the same inode mutex bottle neck.
3. lli_size_sem - I think this can be converted into a reader-writer semaphore. Currently a mutex is taken for each write in order to check and potentially update the inode size. What I think we can do, is first take a read lock to check the inode size, if it needs updating then promote to a write lock. In the normal case, I would imagine most threads would only takea read lock, thus allowing most threads to proceed without blocking here.
|
|
jay, Do you have any ideas as to how we can address the i_mutex bottle neck we're seeing from the generic_file_aio_write call? and/or why that is such an issue for us compared to local filesystems?
|
|
I tried my hand at manipulating the kernel's range locking implementation to do what we need to satisfy (1) from my comment above, but no longer think that's feasible. I've pushed a where I tried to do that here: http://review.whamcloud.com/6320 but I don't think it's fit for landing. We'll probably have to create own range lock implementation and add it into libcfs.
|
|
I spent some time looking into what occurs under the inode mutex, in the context of the generic_file_aio_write call, specifically looking for ways to speed things up; but nothing obvious is jumping out at me. Nothing in particular is jumping out as a blatant time sink, and instead it seems like we just do "too much" in this call path and every little thing adds up to a lot of time. I ran a simple test using dd with ftrace enabled to try and better understand where the time is spent, and have attached the results of that to this ticket in the file "lustre-dd-1M-ftrace.log.bz2".
I've also attached the ftrace results of the same test but writing to a ext4 file system for comparison in "ext4-dd-1M-ftrace.log.bz2". It's clear looking at the ftrace output and the ext4 sources that it is significantly simpler and faster as a result, which is to be expected. It also somewhat explains why ext4 sees much better single shared file performance (~1.1 GB/s)*, despite using the same generic_file_aio_write call path (i.e. serializing on the same inode mutex).
Some timing information gathered from the ftrace log shows most of the time being spent on per page operations, ll_write_begin and ll_write_end specifically. Within ll_write_begin, ll_cl_init appeared to take up much of the time, and within ll_write_end, vvp_io_commit_write appeared to take up much of the time. I copied the timing information of these functions, for the first several pages in the transfer, into the following table:
If we intend to continue using the generic_file_aio_write call, we must speed up the execution time of ll_write_begin and ll_write_end if we are to see better single shared file performance out of a single client. Based on the ftrace data I collected, I'm not sure there is an easy way to do this, and it might take a significant overhaul to the client infrastructure.
With that said, there is talk upstream about removing the inode mutex with a range lock (http://lwn.net/Articles/535843/). I believe this would completely remove our source of contention, but doesn't help us in the short term (or even long term if it doesn't get merged).
- The ext4 test was FIO using a single shared file with 16 threads each writing 1G. A single thread writing 16G test was also run and saw ~1.5 GB/s. The "disk" used for the ext4 FS was a 16 GB ramdisk.
|
|
Hi Prakash,
Can you please share me the options you used for ftrace? I'd like to reproduce the result and tune the performance.
Jinshan
|
|
Sure, here's how I gathered the results:
The results I posted above were taken in a VM, so you might see slightly different results on real hardware, but I'd expect the overall trend to be the same.
|
|
Hi Prakash,
Sorry for delay response. Is it possible for me to write an HLD so that you can work out a patch to solve this problem?
Jinshan
|
|
Yes, please do!
|
|
Hi Prakash, Please check the attachment for design of range lock.
|
|
Jinshan, thanks for the HLD. Just to assure myself, you want to ignore the inode mutex completely by using the lockless __generic_file_aio_write function, right? It seems like a simple first patch can be made to make that change, and atomicity will still be maintained using the current lli_write_lock and the lli_trunc_sem as you described in the HLD. Of course this won't garner any performance benefits, but a follow up patch to break the lli_write_mutex into a range lock would.
|
|
Yes, exactly.
|
|
One tricky thing about inode mutex is to check all references of it in the client code to see if our change will affect it.
|
|
I've pushed a new version of http://review.whamcloud.com/6320 which attempts to add a range lock implementation using Lustre's interval tree implementation. It's still a work in progress, but I wanted to push it for review to get some feedback. Specifically, I seem to be using the waitq interface incorrectly, so any help fixing that would be appreciated.
|
|
Thank you, Prakash, I will take a look.
|
|
Moving this to 2.6 Release version as we have passed feature freeze for 2.5
|
|
Lai
Could you please have a look at these existing patches and see what further effort is required to complete this work?
Thanks
Peter
|
|
I believe the patches are ready - the only effort is to wait for permission and follow our internal quality process to land.
|
|
Patch landed to Master.
|
Generated at Sat Feb 10 01:18:40 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
Range Lock for Write.docx
ext4-dd-1M-ftrace.log.bz2