https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5212 Lustre <p>After upgrading a system from <br/> lustre 2.4.0-1 / zfs-0.6.2-1 to<br/> lustre 2.4.2-1 / zfs-0.6.3-1</p> <p>mdtest shows a signficantly lower stat performance - about 8000 iops vs 14400</p> <p>File reads and file removals are a bit worse, but not as severe. See the attached graph.</p> <p>We do see other marked improvements with the upgrade, for example with system processes waiting on the MDS.</p> <p>I wonder if this is some kind of expected perfromance tradeoff for the new version? I'm guessing the absolute numbers for stat are still acceptable for our workload, but it is quite a large relative difference.</p> <p>Scott</p> centos 6.5 LU-5212

poor stat performance after upgrade from zfs-0.6.2-1/lustre-2.4.0-1 to zfs-0.6.3-1

Bug Minor Resolved Won't Fix WC Triage Scott Nolin llnl prz zfs Tue, 17 Jun 2014 15:49:26 +0000 Thu, 5 Apr 2018 14:03:56 +0000 Thu, 5 Apr 2018 14:03:56 +0000 Lustre 2.4.2 0 8 <p>Hi Scott,<br/> Can you confirm lustre 2.4.2 and zfs 0.6.3?</p> <p>Have you seen this patch <a href="https://jira.hpdd.intel.com/browse/LU-4944" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4944</a> ?</p> <p>Can you collect please the /proc/spl/kstat/zfs/arcstats during the benchmark and upload a graph of:</p> <ul class="alternate" type="square"> <li>arc_meta_limit</li> <li>arc_meta_size</li> <li>c_max</li> <li>c_min</li> <li>p</li> <li>size</li> </ul> <p>thanks</p> <p>This is lustre 2.4.2 and zfs 0.6.3 from the ZFS on Linux EPEL repository</p> <p>I'm not certain I am following this correctly, but the comments on <a href="https://jira.whamcloud.com/browse/LU-4944" title="build fails with latest zfs source" class="issue-link" data-issue-key="LU-4944"><del>LU-4944</del></a> suggest this patch is already in our zfs version</p> <p>I'm not sure when I can run the benchmark again to collect stats while running. Here's limit values at least</p> <p>arc_meta_limit 10000000000<br/> arc_meta_max 10000274368<br/> c_min 4194304<br/> c_max 150000000000</p> <p>When I do get to run this it will be easier if I can just use arcstat.py to collect data. If I do that, which of these fields do you want to see on a graph?</p> <p>Field definitions are as follows:<br/> l2bytes : bytes read per second from the L2ARC<br/> l2hits : L2ARC hits per second<br/> read : Total ARC accesses per second<br/> dmis : Demand Data misses per second<br/> mru : MRU List hits per second<br/> dread : Demand data accesses per second<br/> mread : Metadata accesses per second<br/> c : ARC Target Size<br/> ph% : Prefetch hits percentage<br/> l2hit% : L2ARC access hit percentage<br/> pm% : Prefetch miss percentage<br/> mfu : MFU List hits per second<br/> mm% : Metadata miss percentage<br/> pread : Prefetch accesses per second<br/> miss : ARC misses per second<br/> mrug : MRU Ghost List hits per second<br/> dhit : Demand Data hits per second<br/> mfug : MFU Ghost List hits per second<br/> hits : ARC reads per second<br/> dm% : Demand Data miss percentage<br/> miss% : ARC miss percentage<br/> mhit : Metadata hits per second<br/> dh% : Demand Data hit percentage<br/> mh% : Metadata hit percentage<br/> pmis : Prefetch misses per second<br/> l2asize : Actual (compressed) size of the L2ARC<br/> l2miss% : L2ARC access miss percentage<br/> l2miss : L2ARC misses per second<br/> mmis : Metadata misses per second<br/> phit : Prefetch hits per second<br/> hit% : ARC Hit percentage<br/> eskip : evict_skip per second<br/> arcsz : ARC Size<br/> time : Time<br/> l2read : Total L2ARC accesses per second<br/> l2size : Size of the L2ARC<br/> mtxmis : mutex_miss per second<br/> rmis : recycle_miss per second</p> <p>Hi Scott,<br/> the interesting thing is to collect arcstats every for example 5 seconds and create a graph of the values using for example gnuplot during the test. arcstats.py is more to understand the cache hit/miss.<br/> In my experience your problem is more on optimizing the arc_meta_limit size, but to validate this, I need to see the memory consumption during the run.</p> <p>Have you compiled lustre 2.4.2 against ZFS 0.6.3?</p> <p>I didn't compile it, Brian Behlendorf or whomever built it for zfsonlinux epel repository did.</p> <p>But yes, this is lustre 2.4.2 and zfs 0.6.3 </p> <p>I'll make graphs shortly</p> <p>Scott</p> <p>Attached are requested stats. I broke out the non-constant stats and adjusted a bit to make it a little more interesting.</p> <p>I can't understand nothing... could you please use MB for the y-axis and take a graph only for:<br/> -size<br/> -arc_meta_limit<br/> -arc_meta_size</p> <p>thanks</p> <p>The y-axis is simply the raw data from /proc/spl/kstat/zfs/arcstats - I just put it into a graph quickly, it must be bytes.</p> <p>Note that "arc_meta_size" doesn't exist in arcstats, but there is "meta_size" - I assume the same thing.</p> <p>I will make those graphs in MB for you tomorrow, but here's a description doing some approximate math quickly which should explain it.</p> <p>1) arc_meta_limit is a constant, why graph it? Our value was 10000000000 bytes (aka 1E10) = 9536 MB</p> <p>2) arc_meta_size - does not exist, assuming we want meta_size - this is in 'arcstat-2.png':<br/> Ranges from about 6.31E9 to 6.95E9 bytes = 6017 MB to 6628 MB</p> <p>3) Size - also in arcstat-2.png<br/> 9.5E9 to 1.03E10 bytes = 9059 MB to 9822MB</p> <p>I ran 2 iterations of the mdtest, and I think you can see the test finish and restart in that graph.</p> <p>I captured all the stats, so whatever graphs in whatever format might help, I'll make tomorrow.</p> <p>Thanks,<br/> Scott</p> <p>Scott,</p> <p>In the 1st graph, p (i.e. ARC adaptation parameter) almost never changed, which was weird. Can you verify from your data that p never really changed or its changes were too small to be seen on the graph?</p> <p>Also, can you make sure you have this patch <a href="http://review.whamcloud.com/#/c/10237/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10237/</a> on the server? It was just landed and in my opinion it makes a lot of sense to apply that patch before trying to tune the ARC.</p> <p>Isaac,</p> <p>Regarding 'p' in the first graph - you just can't see it due to the scale. See the 'arcstat-3.png' graph which shows 'p' on it's own changing.</p> <p>Regarding the patch (I see it's now also here - <a href="https://jira.whamcloud.com/browse/LU-5164" title="Limit lu_object cache (ZFS and osd-zfs)" class="issue-link" data-issue-key="LU-5164"><del>LU-5164</del></a>) - no we don't have that patch applied. However, we also didn't have it applied with zfs-0.6.2/lustre-2.4.0 and with the new version stat performance has regressed. ZFS arc performance in general though seems much better.</p> <p>Scott</p> <p>arcstat-MB.png shows meta-size, size, and arc-meta-limit in MB.</p> <p>We have upgraded a second filesystem with similar resources on the MDS/MDT, and see pretty much the same performance difference for stat in mdtest.</p> <p>Scott</p> <p>Do you have 32GB of RAM? if yes could you set these parameters:</p> <ol> <li>cat /etc/modprobe.d/zfs.conf<br/> options zfs zfs_arc_max=30687091200<br/> options zfs zfs_arc_meta_limit=25687091200</li> </ol> <p>reboot your system and collect data?<br/> could you also add the modinfo zfs output?<br/> thanks</p> <p>We have 256GB of RAM</p> <p>modinfo zfs output -</p> <p>filename: /lib/modules/2.6.32-358.6.2.el6.x86_64/extra/zfs.ko<br/> version: 0.6.3-1<br/> license: CDDL<br/> author: Sun Microsystems/Oracle, Lawrence Livermore National Laboratory<br/> description: ZFS<br/> srcversion: C29A443E3D2B93F605A540B<br/> depends: spl,znvpair,zcommon,zunicode,zavl<br/> vermagic: 2.6.32-358.6.2.el6.x86_64 SMP mod_unload modversions<br/> parm: zvol_inhibit_dev:Do not create zvol device nodes (uint)<br/> parm: zvol_major:Major number for zvol device (uint)<br/> parm: zvol_threads:Number of threads for zvol device (uint)<br/> parm: zvol_max_discard_blocks:Max number of blocks to discard (ulong)<br/> parm: zio_injection_enabled:Enable fault injection (int)<br/> parm: zio_bulk_flags:Additional flags to pass to bulk buffers (int)<br/> parm: zio_delay_max:Max zio millisec delay before posting event (int)<br/> parm: zio_requeue_io_start_cut_in_line:Prioritize requeued I/O (int)<br/> parm: zfs_sync_pass_deferred_free:Defer frees starting in this pass (int)<br/> parm: zfs_sync_pass_dont_compress:Don't compress starting in this pass (int)<br/> parm: zfs_sync_pass_rewrite:Rewrite new bps starting in this pass (int)<br/> parm: zil_replay_disable:Disable intent logging replay (int)<br/> parm: zfs_nocacheflush:Disable cache flushes (int)<br/> parm: zil_slog_limit:Max commit bytes to separate log device (ulong)<br/> parm: zfs_read_chunk_size:Bytes to read per chunk (long)<br/> parm: zfs_immediate_write_sz:Largest data block to write to zil (long)<br/> parm: zfs_flags:Set additional debugging flags (int)<br/> parm: zfs_recover:Set to attempt to recover from fatal errors (int)<br/> parm: zfs_expire_snapshot:Seconds to expire .zfs/snapshot (int)<br/> parm: zfs_vdev_aggregation_limit:Max vdev I/O aggregation size (int)<br/> parm: zfs_vdev_read_gap_limit:Aggregate read I/O over gap (int)<br/> parm: zfs_vdev_write_gap_limit:Aggregate write I/O over gap (int)<br/> parm: zfs_vdev_max_active:Maximum number of active I/Os per vdev (int)<br/> parm: zfs_vdev_async_write_active_max_dirty_percent:Async write concurrency max threshold (int)<br/> parm: zfs_vdev_async_write_active_min_dirty_percent:Async write concurrency min threshold (int)<br/> parm: zfs_vdev_async_read_max_active:Max active async read I/Os per vdev (int)<br/> parm: zfs_vdev_async_read_min_active:Min active async read I/Os per vdev (int)<br/> parm: zfs_vdev_async_write_max_active:Max active async write I/Os per vdev (int)<br/> parm: zfs_vdev_async_write_min_active:Min active async write I/Os per vdev (int)<br/> parm: zfs_vdev_scrub_max_active:Max active scrub I/Os per vdev (int)<br/> parm: zfs_vdev_scrub_min_active:Min active scrub I/Os per vdev (int)<br/> parm: zfs_vdev_sync_read_max_active:Max active sync read I/Os per vdev (int)<br/> parm: zfs_vdev_sync_read_min_active:Min active sync read I/Os per vdev (int)<br/> parm: zfs_vdev_sync_write_max_active:Max active sync write I/Os per vdev (int)<br/> parm: zfs_vdev_sync_write_min_active:Min active sync write I/Osper vdev (int)<br/> parm: zfs_vdev_mirror_switch_us:Switch mirrors every N usecs (int)<br/> parm: zfs_vdev_scheduler:I/O scheduler (charp)<br/> parm: zfs_vdev_cache_max:Inflate reads small than max (int)<br/> parm: zfs_vdev_cache_size:Total size of the per-disk cache (int)<br/> parm: zfs_vdev_cache_bshift:Shift size to inflate reads too (int)<br/> parm: zfs_txg_timeout:Max seconds worth of delta per txg (int)<br/> parm: zfs_read_history:Historic statistics for the last N reads (int)<br/> parm: zfs_read_history_hits:Include cache hits in read history (int)<br/> parm: zfs_txg_history:Historic statistics for the last N txgs (int)<br/> parm: zfs_deadman_synctime_ms:Expiration time in milliseconds (ulong)<br/> parm: zfs_deadman_enabled:Enable deadman timer (int)<br/> parm: spa_asize_inflation:SPA size estimate multiplication factor (int)<br/> parm: spa_config_path:SPA config file (/etc/zfs/zpool.cache) (charp)<br/> parm: zfs_autoimport_disable:Disable pool import at module load (int)<br/> parm: metaslab_debug_load:load all metaslabs during pool import (int)<br/> parm: metaslab_debug_unload:prevent metaslabs from being unloaded (int)<br/> parm: zfs_zevent_len_max:Max event queue length (int)<br/> parm: zfs_zevent_cols:Max event column width (int)<br/> parm: zfs_zevent_console:Log events to the console (int)<br/> parm: zfs_top_maxinflight:Max I/Os per top-level (int)<br/> parm: zfs_resilver_delay:Number of ticks to delay resilver (int)<br/> parm: zfs_scrub_delay:Number of ticks to delay scrub (int)<br/> parm: zfs_scan_idle:Idle window in clock ticks (int)<br/> parm: zfs_scan_min_time_ms:Min millisecs to scrub per txg (int)<br/> parm: zfs_free_min_time_ms:Min millisecs to free per txg (int)<br/> parm: zfs_resilver_min_time_ms:Min millisecs to resilver per txg (int)<br/> parm: zfs_no_scrub_io:Set to disable scrub I/O (int)<br/> parm: zfs_no_scrub_prefetch:Set to disable scrub prefetching (int)<br/> parm: zfs_dirty_data_max_percent:percent of ram can be dirty (int)<br/> parm: zfs_dirty_data_max_max_percent:zfs_dirty_data_max upper bound as % of RAM (int)<br/> parm: zfs_delay_min_dirty_percent:transaction delay threshold (int)<br/> parm: zfs_dirty_data_max:determines the dirty space limit (ulong)<br/> parm: zfs_dirty_data_max_max:zfs_dirty_data_max upper bound in bytes (ulong)<br/> parm: zfs_dirty_data_sync:sync txg when this much dirty data (ulong)<br/> parm: zfs_delay_scale:how quickly delay approaches infinity (ulong)<br/> parm: zfs_prefetch_disable:Disable all ZFS prefetching (int)<br/> parm: zfetch_max_streams:Max number of streams per zfetch (uint)<br/> parm: zfetch_min_sec_reap:Min time before stream reclaim (uint)<br/> parm: zfetch_block_cap:Max number of blocks to fetch at a time (uint)<br/> parm: zfetch_array_rd_sz:Number of bytes in a array_read (ulong)<br/> parm: zfs_pd_blks_max:Max number of blocks to prefetch (int)<br/> parm: zfs_send_corrupt_data:Allow sending corrupt data (int)<br/> parm: zfs_mdcomp_disable:Disable meta data compression (int)<br/> parm: zfs_nopwrite_enabled:Enable NOP writes (int)<br/> parm: zfs_dedup_prefetch:Enable prefetching dedup-ed blks (int)<br/> parm: zfs_dbuf_state_index:Calculate arc header index (int)<br/> parm: zfs_arc_min:Min arc size (ulong)<br/> parm: zfs_arc_max:Max arc size (ulong)<br/> parm: zfs_arc_meta_limit:Meta limit for arc size (ulong)<br/> parm: zfs_arc_meta_prune:Bytes of meta data to prune (int)<br/> parm: zfs_arc_grow_retry:Seconds before growing arc size (int)<br/> parm: zfs_arc_p_aggressive_disable:disable aggressive arc_p grow (int)<br/> parm: zfs_arc_p_dampener_disable:disable arc_p adapt dampener (int)<br/> parm: zfs_arc_shrink_shift:log2(fraction of arc to reclaim) (int)<br/> parm: zfs_disable_dup_eviction:disable duplicate buffer eviction (int)<br/> parm: zfs_arc_memory_throttle_disable:disable memory throttle (int)<br/> parm: zfs_arc_min_prefetch_lifespan:Min life of prefetch block (int)<br/> parm: l2arc_write_max:Max write bytes per interval (ulong)<br/> parm: l2arc_write_boost:Extra write bytes during device warmup (ulong)<br/> parm: l2arc_headroom:Number of max device writes to precache (ulong)<br/> parm: l2arc_headroom_boost:Compressed l2arc_headroom multiplier (ulong)<br/> parm: l2arc_feed_secs:Seconds between L2ARC writing (ulong)<br/> parm: l2arc_feed_min_ms:Min feed interval in milliseconds (ulong)<br/> parm: l2arc_noprefetch:Skip caching prefetched buffers (int)<br/> parm: l2arc_nocompress:Skip compressing L2ARC buffers (int)<br/> parm: l2arc_feed_again:Turbo L2ARC warmup (int)<br/> parm: l2arc_norw:No reads during writes (int)</p> <p>Gabriele,</p> <p>I'm working on this filesystem with Scott:</p> <p>Speaking to the Arc Cache settings, we are currently using:</p> <p>options zfs zfs_arc_meta_limit=10000000000<br/> options zfs zfs_arc_max=150000000000</p> <p>However, we're nowhere near to filling that up so we're not seeing any excessive cache pressure right now.</p> <p>Have you set these values?<br/> by default arc_meta_limt should be 1/4 of the total RAM, so in your case 64GB and the zfs_arc_max 1/2.</p> <p>In the arcstat-MB seems to be 10GB only.</p> <p>could you increase these value? also the zfs_arc_max?</p> <p>Yes, we set these values after observations of different values on ZFS 0.6.2. The larger meta_limit helped us avoid running into ugly cache issues.</p> <p>Either way, the cache is only using about 10GB right now as the filesystem has only been up for two days and is relatively quiet. We can't force it to use more of the cache without more activity.</p> <p>Okay but yesterday Scott captured these values:<br/> arc_meta_limit 10000000000<br/> arc_meta_max 10000274368</p> <p>and for me you go out of the arc memory for metadata during your mdtest.</p> <p>Sorry about that, we were missing a 0 on the arc_meta_limit. We'll retest with the new values.</p> <p>I've completed an initial run with the zfs_arc_meta_limit and max set more approprately to 100G and 150G.</p> <p>The mdtest data doesn't look any better, it's actually a bit worse. Complicating things, the filesystem is now in use by other jobs, not heavy use but a few hundred iops on various tasks it looks like (just watching jobstats in general). I'll post actual data if I can get another test done.</p> <p>Regardless of that one, our <b>other</b> filesystem has more appropriate numbers to start with (it was left at default), and see a very similar difference in stat performance. So I don't really expect much...</p> <p>Scott</p> <p>Here are results with more appropriate limits, included the graph with arc_meta_limit to show we're not exceeding it, and also a version with better scale.</p> <p>Notice how things certainly aren't better, they're a little worse.</p> <p>These graphs all have "95G" in the title. I wish I could embed images within comments to make it flow better.</p> <p>I don't mean to barge in so late to the party, but I'm curious what the status of this issue is? It's a little hard to follow the comments and the attached graphs. What's the observed performance degradation, and what configuration changes/experiments have been tried (and what were the results)?</p> <p>The observed performance degradation is ~45% lower stat IOPs in mdtest (8000 vs 14400) after the zfs/lustre upgrade. </p> <p>We adjusted the arc_meta_limit (as it was set poorly) but it made no difference. The easiest graph to look at is this one: <a href="https://jira.hpdd.intel.com/secure/attachment/15192/mdtest-zfs063-95G-arc_meta_limit.png" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/15192/mdtest-zfs063-95G-arc_meta_limit.png</a></p> <p>The graphs are all for one particular filesystem, but we have a second file system with similar hardware and the same software versions and saw similar degradation in stat performance for mdtest.</p> <p>If anyone runs mdtest just prior to upgrading lustre/zfs I'd be interested in their results, I suspect it will be similar. Our software is from the zfsonlinux repo with no additional patches.</p> <p>Scott</p> <p>Thanks Scott. That definitely doesn't sit well with me.</p> <p>Can you post the command you used as a test? Do you have the exact mdtest command/options you used? How many nodes? If I can get some time, I might try and reproduce this and see if I can better understand what's going on here. It's definitely <b>not</b> expected nor desired for the performance to drop like that; I want to get to the bottom of this.</p> <p>Also, what's the Y axis label in the graph you linked to? I saw that earlier, but I can't make sense of it without labels. My initial interpretation was the Y axis is seconds, but that would mean lower is better, which doesn't agree with the claim of a performance decrease.</p> <blockquote> <p>Also, what's the Y axis label in the graph you linked to? I saw that earlier, but I can't make sense of it without labels. My initial interpretation was the Y axis is seconds, but that would mean lower is better, which doesn't agree with the claim of a performance decrease.</p></blockquote> <p>Actually, I think I got it now. The Y axis must be the rate of operations per second, which lines up with your claim of 14400 stat/s prior and 8000 stat/s now.</p> <p>When you get a chance, please update us with the command used to generate the workload.</p> <p>Y-axis is IOPs.</p> <p>The command info:</p> <p>mdtest-1.9.1 was launched with 64 total task(s) on 4 node(s)<br/> Command line used: /home/scottn/benchmarks/mdtest -i 2 -F -n 4000 -d /arcdata/scottn/mdtest</p> <p>64 tasks, 256000 files</p> <p>Scott</p> <p>I would also add, that while this absolute number from mdtest is worse, in use so far the upgrade has been an improvement. Performance doesn't seem to degrade so quickly with file creates, and things like interactive 'ls -l' are much better.</p> <p>Scott</p> <blockquote> <p>I would also add, that while this absolute number from mdtest is worse, in use so far the upgrade has been an improvement. Performance doesn't seem to degrade so quickly with file creates, and things like interactive 'ls -l' are much better.</p></blockquote> <p>Glad to hear it! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'm still a bit puzzled regarding the stat's though. I'm going to try and reproduce this using our test cluster; stay tuned.</p> <p>Interesting.. I think I see similar reduced performance with stats as well.. Hm..</p> <p>So here's the mdtest output with releases based on lustre 2.4.2 and zfs 0.6.3 on the servers:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>hype355@root:srun -- mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 -- started at 07/09/2014 13:09:02 -- mdtest-1.8.3 was launched with 64 total task(s) on 64 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 Path: /p/lcratery/surya1/LU-5212 FS: 1019.6 TiB Used FS: 50.3% Inodes: 866.3 Mi Used Inodes: 60.2% 64 tasks, 256000 files SUMMARY: (of 8 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 2046.438 838.703 1534.565 375.574 File stat : 65205.403 23577.494 57837.499 13089.055 File removal : 4780.471 4647.670 4719.076 45.088 Tree creation : 505.051 34.332 221.404 196.950 Tree removal : 12.423 10.049 11.123 0.763 -- finished at 07/09/2014 13:40:52 -- </pre> </div></div> <p>And here's the mdtest output with releases based on lustre 2.4.0 and zfs 0.6.2 on the servers:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>hype355@root:srun -- mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 -- started at 07/09/2014 14:43:06 -- mdtest-1.8.3 was launched with 64 total task(s) on 64 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-1 Path: /p/lcratery/surya1/LU-5212 FS: 1019.6 TiB Used FS: 50.3% Inodes: 861.8 Mi Used Inodes: 60.5% 64 tasks, 256000 files SUMMARY: (of 8 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 1627.029 810.017 1320.848 239.655 File stat : 99560.417 69839.184 88798.194 9632.641 File removal : 4352.713 3279.728 4029.607 413.213 Tree creation : 348.675 33.174 194.944 141.913 Tree removal : 15.176 10.103 12.088 1.386 -- finished at 07/09/2014 15:19:02 -- </pre> </div></div> <p>Which shows about a 34% decrease in the mean "File stat" performance with the lustre 2.4.2 and zfs 0.6.3 release (I'm assuming the number reported is operations per second). That's no good.</p> <p>Scott, can you try increasing the `lu_cache_nr` module option and re-running the test?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># zwicky-lcy-mds1 /root &gt; cat /sys/module/obdclass/parameters/lu_cache_nr 256 </pre> </div></div> <p>Try increasing it to something much larger, maybe 1M. I'd try that myself, but our testing resource is busy with other work at the moment.</p> <p>Prakash, we will give this a try soon.</p> <p>Scott</p> <p>Scott, I was able to squeeze in a test run with <tt>lu_cache_nr=1048576</tt> on the MDS and all OSS nodes in the filesystem. I didn't see any significant difference:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>hype355@root:srun -- mdtest -v -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-5 -- started at 07/16/2014 09:23:46 -- mdtest-1.8.3 was launched with 64 total task(s) on 64 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -v -i 8 -F -n 4000 -d /p/lcratery/surya1/LU-5212/mdtest-5 Path: /p/lcratery/surya1/LU-5212 FS: 1019.6 TiB Used FS: 50.3% Inodes: 834.8 Mi Used Inodes: 62.5% 64 tasks, 256000 files SUMMARY: (of 8 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 3060.802 2525.669 2719.003 161.410 File stat : 72310.501 32382.555 57016.755 11440.553 File removal : 4344.489 4043.991 4224.141 97.727 Tree creation : 377.644 32.784 147.864 126.958 Tree removal : 11.800 9.356 10.626 0.884 -- finished at 07/16/2014 09:45:06 -- </pre> </div></div> <p>Prakash, thanks for letting me know. We won't bother running it then. Ours is a production cluster, we can typically run these tests though as it's not heavily used all the time, but it's not easy.</p> <p>Scott</p> <p>I imagine the performance is quite different on more current versions of Lustre and ZFS</p> Related LU-5203 Development Epic/Theme Performance zfs Rank 1|hzwp8f: Rank (Obsolete) 14543 Severity