Details
-
Improvement
-
Resolution: Fixed
-
Major
-
None
-
None
-
11351
Description
Testing has shown that a larger MDT journal size can increase performance significantly now that SMP scaling allows the MDT code to perform more operations per second. I'd like to increase the default journal size for newly formatted MDTs.
Performance test results shown with a 4GB journal size on the following hardware:
Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz IVY BRIDGE 128 GB RAM 6x INTEL SSD S3700 400GB 1x Intel TrueScale Card QDR 2x Intel 82599EB 10-Gigabit SFI/SFP+ 4x Intel I350 Gigabit CentOS 6.4 Lustre 2.4.1 + CLIO simplification patch
Attachments
Issue Links
- is related to
-
LU-4611 too many transaction credits (32279 > 25600)
-
- Resolved
-
Activity
Using your patch I also increased the OST limit to 4GB. Collecting data.
James, no testing has been done for the OST create rate yet. The patch is only changing the journal size limit for the MDT so far. We could assume the OST create rate with the 400MB journal is approximately the same as the MDT with 400MB journal (about 1/2 of 4GB journal) so if there are at least twice as many OSTs as MDTs the object create rate should be enough.
It would be great to verify actual OST object create performance, since OST object create is single threaded, and it may not be able to keep up with hundreds of MDS create threads running in parallel. In theory it is possible to run echo_client on the OST device to create objects, but this would probably not exercise the batches create that the MDS uses. It is probably worthwhile to file a separate bug for implementing any needed infrastructure and testing this.
Hi Andreas,
on the same hardware, with the same configuration for mrs-survey (32 threads, 32 directories, 400K files each dirs), I have run other benchmarks.
I'm using the 2.3.11 lustre server version included in IEEL 1.0 and not 2.4.1 as in the previous experiment.
Patch 8111 has landed for 2.6.0 so the MDTs will now have 4GB journals by default, if the MDT is large enough.
I also note that http://review.whamcloud.com/9258 will reduce the credit reservation for create operations, so it might avoid the need for such a large journal and/or improve performance further.
James, did you get an test results with OSTs and 4GB journals? If not, I'm inclined to close this bug and you can open a separate bug to track such a change.