Details
-
Bug
-
Resolution: Won't Fix
-
Minor
-
None
-
Lustre 2.4.2
-
centos 6.5
-
3
-
14543
Description
After upgrading a system from
lustre 2.4.0-1 / zfs-0.6.2-1 to
lustre 2.4.2-1 / zfs-0.6.3-1
mdtest shows a signficantly lower stat performance - about 8000 iops vs 14400
File reads and file removals are a bit worse, but not as severe. See the attached graph.
We do see other marked improvements with the upgrade, for example with system processes waiting on the MDS.
I wonder if this is some kind of expected perfromance tradeoff for the new version? I'm guessing the absolute numbers for stat are still acceptable for our workload, but it is quite a large relative difference.
Scott
Attachments
Issue Links
- is related to
-
LU-5203 Update ZFS Version to 0.6.3
-
- Resolved
-
Activity
Interesting.. I think I see similar reduced performance with stats as well.. Hm..
So here's the mdtest output with releases based on lustre 2.4.2 and zfs 0.6.3 on the servers:
hype355@root:srun -- mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 -- started at 07/09/2014 13:09:02 -- mdtest-1.8.3 was launched with 64 total task(s) on 64 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 Path: /p/lcratery/surya1/LU-5212 FS: 1019.6 TiB Used FS: 50.3% Inodes: 866.3 Mi Used Inodes: 60.2% 64 tasks, 256000 files SUMMARY: (of 8 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 2046.438 838.703 1534.565 375.574 File stat : 65205.403 23577.494 57837.499 13089.055 File removal : 4780.471 4647.670 4719.076 45.088 Tree creation : 505.051 34.332 221.404 196.950 Tree removal : 12.423 10.049 11.123 0.763 -- finished at 07/09/2014 13:40:52 --
And here's the mdtest output with releases based on lustre 2.4.0 and zfs 0.6.2 on the servers:
hype355@root:srun -- mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 -- started at 07/09/2014 14:43:06 -- mdtest-1.8.3 was launched with 64 total task(s) on 64 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 Path: /p/lcratery/surya1/LU-5212 FS: 1019.6 TiB Used FS: 50.3% Inodes: 861.8 Mi Used Inodes: 60.5% 64 tasks, 256000 files SUMMARY: (of 8 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 1627.029 810.017 1320.848 239.655 File stat : 99560.417 69839.184 88798.194 9632.641 File removal : 4352.713 3279.728 4029.607 413.213 Tree creation : 348.675 33.174 194.944 141.913 Tree removal : 15.176 10.103 12.088 1.386 -- finished at 07/09/2014 15:19:02 --
Which shows about a 34% decrease in the mean "File stat" performance with the lustre 2.4.2 and zfs 0.6.3 release (I'm assuming the number reported is operations per second). That's no good.
I would also add, that while this absolute number from mdtest is worse, in use so far the upgrade has been an improvement. Performance doesn't seem to degrade so quickly with file creates, and things like interactive 'ls -l' are much better.
Glad to hear it! 
I'm still a bit puzzled regarding the stat's though. I'm going to try and reproduce this using our test cluster; stay tuned.
I would also add, that while this absolute number from mdtest is worse, in use so far the upgrade has been an improvement. Performance doesn't seem to degrade so quickly with file creates, and things like interactive 'ls -l' are much better.
Scott
Y-axis is IOPs.
The command info:
mdtest-1.9.1 was launched with 64 total task(s) on 4 node(s)
Command line used: /home/scottn/benchmarks/mdtest -i 2 -F -n 4000 -d /arcdata/scottn/mdtest
64 tasks, 256000 files
Scott
Also, what's the Y axis label in the graph you linked to? I saw that earlier, but I can't make sense of it without labels. My initial interpretation was the Y axis is seconds, but that would mean lower is better, which doesn't agree with the claim of a performance decrease.
Actually, I think I got it now. The Y axis must be the rate of operations per second, which lines up with your claim of 14400 stat/s prior and 8000 stat/s now.
When you get a chance, please update us with the command used to generate the workload.
Thanks Scott. That definitely doesn't sit well with me.
Can you post the command you used as a test? Do you have the exact mdtest command/options you used? How many nodes? If I can get some time, I might try and reproduce this and see if I can better understand what's going on here. It's definitely not expected nor desired for the performance to drop like that; I want to get to the bottom of this.
Also, what's the Y axis label in the graph you linked to? I saw that earlier, but I can't make sense of it without labels. My initial interpretation was the Y axis is seconds, but that would mean lower is better, which doesn't agree with the claim of a performance decrease.
The observed performance degradation is ~45% lower stat IOPs in mdtest (8000 vs 14400) after the zfs/lustre upgrade.
We adjusted the arc_meta_limit (as it was set poorly) but it made no difference. The easiest graph to look at is this one: https://jira.hpdd.intel.com/secure/attachment/15192/mdtest-zfs063-95G-arc_meta_limit.png
The graphs are all for one particular filesystem, but we have a second file system with similar hardware and the same software versions and saw similar degradation in stat performance for mdtest.
If anyone runs mdtest just prior to upgrading lustre/zfs I'd be interested in their results, I suspect it will be similar. Our software is from the zfsonlinux repo with no additional patches.
Scott
I don't mean to barge in so late to the party, but I'm curious what the status of this issue is? It's a little hard to follow the comments and the attached graphs. What's the observed performance degradation, and what configuration changes/experiments have been tried (and what were the results)?
Here are results with more appropriate limits, included the graph with arc_meta_limit to show we're not exceeding it, and also a version with better scale.
Notice how things certainly aren't better, they're a little worse.
These graphs all have "95G" in the title. I wish I could embed images within comments to make it flow better.
I've completed an initial run with the zfs_arc_meta_limit and max set more approprately to 100G and 150G.
The mdtest data doesn't look any better, it's actually a bit worse. Complicating things, the filesystem is now in use by other jobs, not heavy use but a few hundred iops on various tasks it looks like (just watching jobstats in general). I'll post actual data if I can get another test done.
Regardless of that one, our other filesystem has more appropriate numbers to start with (it was left at default), and see a very similar difference in stat performance. So I don't really expect much...
Scott
Scott, can you try increasing the `lu_cache_nr` module option and re-running the test?
Try increasing it to something much larger, maybe 1M. I'd try that myself, but our testing resource is busy with other work at the moment.