https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4257 Lustre <p>Sanger has an interesting test in which they read from the same file from 20 processes. They first run in parallel and then run serially (after flushing cache). Their expected result is that the serial and parallel runs should take about the same amount of time. What they see however is that parallel reads are about 50% slower than serial reads:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">client1# cat readfile.sh #!/bin/sh dd <span class="code-keyword">if</span>=/lustre/scratch110/sanger/jb23/test/delete bs=4M of=/dev/<span class="code-keyword">null</span> client1# <span class="code-keyword">for</span> i in `seq -w 1 20 ` <span class="code-keyword">do</span> (time $LOC/readfile.sh ) &gt; $LOC/results/${i}_out 2&gt;&amp;1 &amp; done </pre> </div></div> <blockquote> <p>In parallel </p> <p>01_out:real 3m36.228s <br/> 02_out:real 3m36.227s <br/> 03_out:real 3m36.226s <br/> 04_out:real 3m36.224s <br/> 05_out:real 3m36.224s <br/> 06_out:real 3m36.224s <br/> 07_out:real 3m36.222s <br/> 08_out:real 3m36.221s <br/> 09_out:real 3m36.228s <br/> 10_out:real 3m36.222s <br/> 11_out:real 3m36.220s <br/> 12_out:real 3m36.220s <br/> 13_out:real 3m36.228s <br/> 14_out:real 3m36.219s <br/> 15_out:real 3m36.217s <br/> 16_out:real 3m36.218s <br/> 17_out:real 3m36.214s <br/> 18_out:real 3m36.214s <br/> 19_out:real 3m36.211s <br/> 20_out:real 3m36.212s </p> <p>A serial read ( I expect all the time to be in the first read ).</p> <p>grep -i real *_serial <br/> 01_out_serial:real 2m31.372s <br/> 02_out_serial:real 0m1.190s <br/> 03_out_serial:real 0m0.654s <br/> 04_out_serial:real 0m0.562s <br/> 05_out_serial:real 0m0.574s <br/> 06_out_serial:real 0m0.570s <br/> 07_out_serial:real 0m0.574s <br/> 08_out_serial:real 0m0.461s <br/> 09_out_serial:real 0m0.456s <br/> 10_out_serial:real 0m0.462s <br/> 11_out_serial:real 0m0.475s <br/> 12_out_serial:real 0m0.473s <br/> 13_out_serial:real 0m0.582s <br/> 14_out_serial:real 0m0.580s <br/> 15_out_serial:real 0m0.569s <br/> 16_out_serial:real 0m0.679s <br/> 17_out_serial:real 0m0.565s <br/> 18_out_serial:real 0m0.573s <br/> 19_out_serial:real 0m0.579s <br/> 20_out_serial:real 0m0.472s </p> <p>And try the same experiment with nfs </p> <p>Serial access. </p> <p>root@farm3-head4:~/tmp/test/results# grep -i real * <br/> results/01_out_serial:real 0m19.923s <br/> results/02_out_serial:real 0m1.373s <br/> results/03_out_serial:real 0m1.237s <br/> results/04_out_serial:real 0m1.276s <br/> results/05_out_serial:real 0m1.289s <br/> results/06_out_serial:real 0m1.297s <br/> results/07_out_serial:real 0m1.265s <br/> results/08_out_serial:real 0m1.278s <br/> results/09_out_serial:real 0m1.224s <br/> results/10_out_serial:real 0m1.225s <br/> results/11_out_serial:real 0m1.221s <br/> ...</p></blockquote> <p>So the question is:<br/> Why is the access slower if we are accessing the file in parallel and it is not in the cache ?</p> <p>Is there some lock contention going on with multiple readers? Or is the Lustre client sending multiple RPCs for the same data, even though there is already an outstanding request? They have tried this on 1.8.x clients as well as 2.5.0. </p> <p>Thanks.</p> LU-4257

parallel dds are slower than serial dds

Bug Minor Resolved Fixed Jinshan Xiong Shuichi Ihara Fri, 15 Nov 2013 15:43:16 +0000 Tue, 16 Jun 2020 09:00:53 +0000 Fri, 3 Jun 2016 20:47:39 +0000 Lustre 2.5.0 Lustre 2.9.0 0 29 <p>Bobijam</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>How large are the files being tested? One stripe or multiple stripes per file? It surprises me that Lustre is slower than NFS, but I guess that isn't directly relevant for the problem itself.</p> <p>It would be useful to collect +rpctrace +dlmtrace +vfstrace +reada debug logs on the client for the parallel and serial runs. That would allow us to see if there are duplicate RPCs being sent. This would be most useful with a 2.5.0 client. It may be that Jinshan's client-side single-threaded performance patches would help improve the overall performance, but it isn't totally clear if they would affect the parallel vs. serial behaviour.</p> <p>They are 2GB, single-striped files. The client nodes have 4GB of RAM and are mounting with the localflock option. </p> <p>I'll work on getting the debug logs.</p> <p>It seems really terrible performance to be reading a 2GB file in 151s (=13.5MB/s) in the serial case, or even 216s (=9.5MB/s) in the parallel case. It seems like something else is broken in this filesystem that is causing everything to run slowly? Even my 8-year-old home system runs this fast.</p> <p>Note when I filtered on +rpctrace +dlmtrace +vfstrace +reada I got no events so I have given the full logs....</p> <p>Sorry.</p> <p>This is using my desktop which is a lustre 2.5.1 client. </p> <p>root@deskpro21498:/tmp/results# grep copied /tmp/results/<b>ser</b><br/> /tmp/results/01_out_ser:2147483648 bytes (2.1 GB) copied, 19.4224 s, 111 MB/s<br/> /tmp/results/02_out_ser:2147483648 bytes (2.1 GB) copied, 1.1007 s, 2.0 GB/s<br/> /tmp/results/03_out_ser:2147483648 bytes (2.1 GB) copied, 1.03958 s, 2.1 GB/s<br/> /tmp/results/04_out_ser:2147483648 bytes (2.1 GB) copied, 1.03101 s, 2.1 GB/s<br/> /tmp/results/05_out_ser:2147483648 bytes (2.1 GB) copied, 1.03045 s, 2.1 GB/s<br/> /tmp/results/06_out_ser:2147483648 bytes (2.1 GB) copied, 1.03191 s, 2.1 GB/s<br/> /tmp/results/07_out_ser:2147483648 bytes (2.1 GB) copied, 1.0305 s, 2.1 GB/s<br/> /tmp/results/08_out_ser:2147483648 bytes (2.1 GB) copied, 1.03164 s, 2.1 GB/s<br/> /tmp/results/09_out_ser:2147483648 bytes (2.1 GB) copied, 1.03211 s, 2.1 GB/s<br/> /tmp/results/10_out_ser:2147483648 bytes (2.1 GB) copied, 1.0314 s, 2.1 GB/s<br/> /tmp/results/11_out_ser:2147483648 bytes (2.1 GB) copied, 1.03315 s, 2.1 GB/s<br/> /tmp/results/12_out_ser:2147483648 bytes (2.1 GB) copied, 1.03305 s, 2.1 GB/s<br/> /tmp/results/13_out_ser:2147483648 bytes (2.1 GB) copied, 1.03047 s, 2.1 GB/s<br/> /tmp/results/14_out_ser:2147483648 bytes (2.1 GB) copied, 1.03324 s, 2.1 GB/s<br/> /tmp/results/15_out_ser:2147483648 bytes (2.1 GB) copied, 1.03284 s, 2.1 GB/s<br/> /tmp/results/16_out_ser:2147483648 bytes (2.1 GB) copied, 1.03156 s, 2.1 GB/s<br/> /tmp/results/17_out_ser:2147483648 bytes (2.1 GB) copied, 1.03013 s, 2.1 GB/s<br/> /tmp/results/18_out_ser:2147483648 bytes (2.1 GB) copied, 1.03125 s, 2.1 GB/s<br/> /tmp/results/19_out_ser:2147483648 bytes (2.1 GB) copied, 1.03179 s, 2.1 GB/s<br/> /tmp/results/20_out_ser:2147483648 bytes (2.1 GB) copied, 1.02893 s, 2.1 GB/s<br/> root@deskpro21498:/tmp/results# grep copied /tmp/results/<b>par</b><br/> /tmp/results/01_out_par:2147483648 bytes (2.1 GB) copied, 28.5889 s, 75.1 MB/s<br/> /tmp/results/02_out_par:2147483648 bytes (2.1 GB) copied, 28.535 s, 75.3 MB/s<br/> /tmp/results/03_out_par:2147483648 bytes (2.1 GB) copied, 28.5606 s, 75.2 MB/s<br/> /tmp/results/04_out_par:2147483648 bytes (2.1 GB) copied, 28.5644 s, 75.2 MB/s<br/> /tmp/results/05_out_par:2147483648 bytes (2.1 GB) copied, 28.5557 s, 75.2 MB/s<br/> /tmp/results/06_out_par:2147483648 bytes (2.1 GB) copied, 28.5399 s, 75.2 MB/s<br/> /tmp/results/07_out_par:2147483648 bytes (2.1 GB) copied, 28.5578 s, 75.2 MB/s<br/> /tmp/results/08_out_par:2147483648 bytes (2.1 GB) copied, 28.5544 s, 75.2 MB/s<br/> /tmp/results/09_out_par:2147483648 bytes (2.1 GB) copied, 28.5705 s, 75.2 MB/s<br/> /tmp/results/10_out_par:2147483648 bytes (2.1 GB) copied, 28.5436 s, 75.2 MB/s<br/> /tmp/results/11_out_par:2147483648 bytes (2.1 GB) copied, 28.5596 s, 75.2 MB/s<br/> /tmp/results/12_out_par:2147483648 bytes (2.1 GB) copied, 28.5437 s, 75.2 MB/s<br/> /tmp/results/13_out_par:2147483648 bytes (2.1 GB) copied, 28.5611 s, 75.2 MB/s<br/> /tmp/results/14_out_par:2147483648 bytes (2.1 GB) copied, 28.5543 s, 75.2 MB/s<br/> /tmp/results/15_out_par:2147483648 bytes (2.1 GB) copied, 28.5604 s, 75.2 MB/s<br/> /tmp/results/16_out_par:2147483648 bytes (2.1 GB) copied, 28.5546 s, 75.2 MB/s<br/> /tmp/results/17_out_par:2147483648 bytes (2.1 GB) copied, 28.5358 s, 75.3 MB/s<br/> /tmp/results/18_out_par:2147483648 bytes (2.1 GB) copied, 28.5552 s, 75.2 MB/s<br/> /tmp/results/19_out_par:2147483648 bytes (2.1 GB) copied, 28.5478 s, 75.2 MB/s<br/> /tmp/results/20_out_par:2147483648 bytes (2.1 GB) copied, 28.5414 s, 75.2 MB/s</p> <p>Can I add any information ?</p> <p>Can the problem be reproduced else where ?</p> <p>Hi James,</p> <p>Can you confirm how large the original file was? Also can you double check your debug settings? When I run it on a Lustre 2.1.5 VM I get a ton of debug data. </p> <p>sysctl lnet.debug="+rpctrace +dlmtrace +vfstrace +reada"<br/> lnet.debug = ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace reada config console</p> <p>Here's the script I used:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash lctl clear lctl debug_daemon start /scratch/debug_file 2048 echo 3 &gt; /proc/sys/vm/drop_caches lctl mark parallel read start <span class="code-keyword">for</span> x in 1 2 3 4 5; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=/lustre/pfs/client/testfile bs=4M of=/dev/<span class="code-keyword">null</span> &amp; done wait lctl mark parallel read done echo 3 &gt; /proc/sys/vm/drop_caches lctl mark serial read start <span class="code-keyword">for</span> x in 1 2 3 4 5; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=/lustre/pfs/client/testfile bs=4M of=/dev/<span class="code-keyword">null</span> ; done lctl mark serial read done lctl debug_daemon stop lctl df /scratch/debug_file &gt; /scratch/debug_file.out </pre> </div></div> <p>I was able to reproduce the slow down you mentioned. I'll attach the debug logs that I got. </p> <p>parallel:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.40159 s, 59.6 MB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.40117 s, 59.6 MB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.39914 s, 59.6 MB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.39789 s, 59.6 MB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.39852 s, 59.6 MB/s </pre> </div></div> <p>serial:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 4.57006 s, 57.4 MB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 0.0536567 s, 4.9 GB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 0.046017 s, 5.7 GB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 0.0454 s, 5.8 GB/s 62+1 records in 62+1 records out 262144000 bytes (262 MB) copied, 0.0453329 s, 5.8 GB/s </pre> </div></div> <p>In the most recent results with the 2.5.0 client (we don't have a 2.5.1 yet?) the parallel reads take only 28.5s, while the serial reads take 39s in total, so the parallel reads are faster in aggregate than serial reads, though the single serial read is faster (19.4s) than any of the parallel reads. These results also are more in line with NFS (19.9s for the first serial read, 1.3s for each later read).</p> <p>I updated the script a bit and ran it on my 2.4.0 x86_64 2-core 4-thread client, 2.4.1 server:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/sh FILE=/myth/tmp/100M # previously created OST=myth-OST0000 lctl set_param ldlm.namespaces.$OST*.lru_size=clear lctl set_param llite.*.stats=0 osc.$OST*.rpc_stats=0 <span class="code-keyword">for</span> i in $(seq -w 1 20); <span class="code-keyword">do</span> (time dd <span class="code-keyword">if</span>=$FILE bs=4M of=/dev/<span class="code-keyword">null</span> ) &gt; /tmp/par_${i}.out 2&gt;&amp;1 &amp; done wait lctl get_param llite.*.stats osc.$OST*.rpc_stats | tee /tmp/par_stats.out lctl set_param ldlm.namespaces.$OST*.lru_size=clear lctl set_param llite.*.stats=0 osc.$OST*.rpc_stats=0 <span class="code-keyword">for</span> i in $(seq -w 1 20); <span class="code-keyword">do</span> (time dd <span class="code-keyword">if</span>=$FILE bs=4M of=/dev/<span class="code-keyword">null</span> ) &gt; /tmp/ser_${i}.out 2&gt;&amp;1 done lctl get_param llite.*.stats osc.$OST*.rpc_stats | tee /tmp/par_stats.out </pre> </div></div> <p>This produced the following from rpc_stats, which showed that there are not multiple RPCs being sent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SERIAL read write pages per rpc rpcs % cum % | rpcs % cum % 64: 2 1 1 | 0 0 0 256: 100 98 100 | 0 0 0 PARALLEL read write pages per rpc rpcs % cum % | rpcs % cum % 256: 100 100 100 | 0 0 0 </pre> </div></div> <p>The time profile is as I would expect, shorter single time for serial but longer total time (no parallelism), and a shorter total time for parallel (runtime happens in parallel):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre></pre> </div></div> <p>SERIAL<br/> real 0m9.657s sys 0m1.556s<br/> real 0m0.284s sys 0m0.281s<br/> real 0m0.244s sys 0m0.241s<br/> real 0m0.377s sys 0m0.368s<br/> real 0m0.202s sys 0m0.201s<br/> real 0m0.207s sys 0m0.195s<br/> real 0m0.200s sys 0m0.196s<br/> real 0m0.210s sys 0m0.209s<br/> real 0m0.198s sys 0m0.197s<br/> real 0m0.197s sys 0m0.194s<br/> real 0m0.198s sys 0m0.196s<br/> real 0m0.202s sys 0m0.198s<br/> real 0m0.295s sys 0m0.291s<br/> real 0m0.164s sys 0m0.162s<br/> real 0m0.163s sys 0m0.162s<br/> real 0m0.162s sys 0m0.160s<br/> real 0m0.162s sys 0m0.160s<br/> real 0m0.162s sys 0m0.159s<br/> real 0m0.164s sys 0m0.160s<br/> real 0m0.162s sys 0m0.160s</p> <p>PARALLEL<br/> real 0m12.817s sys 0m0.918s<br/> real 0m12.816s sys 0m1.018s<br/> real 0m12.804s sys 0m0.837s<br/> real 0m12.791s sys 0m0.865s<br/> real 0m12.783s sys 0m0.922s<br/> real 0m12.768s sys 0m0.866s<br/> real 0m12.773s sys 0m0.888s<br/> real 0m12.773s sys 0m0.968s<br/> real 0m12.761s sys 0m0.842s<br/> real 0m12.741s sys 0m0.897s<br/> real 0m12.734s sys 0m0.856s<br/> real 0m12.718s sys 0m0.869s<br/> real 0m12.700s sys 0m0.896s<br/> real 0m12.711s sys 0m0.933s<br/> real 0m12.685s sys 0m0.931s<br/> real 0m12.656s sys 0m0.854s<br/> real 0m12.669s sys 0m0.970s<br/> real 0m12.663s sys 0m0.848s<br/> real 0m12.655s sys 0m0.947s<br/> real 0m12.649s sys 0m0.880s</p> <p>Just as a note, I think I can see the problem however I am busy upgrading a lustre file system... and I want to ensure that my communication is clear, apologies for the delay.</p> <p>I have a basic C program to do a while(read()) style loop until EOF on an ~800Mb file with varying buffer sizes. I tested this on a lustre, tmpfs and nfs using a mix of block sizes and 1, 4 and 32 concurrent copies. For example on lustre with 4 copies:</p> <p>jkb@sf-7-1-02<span class="error">&#91;work/benchmarks&#93;</span> fn=/lustre/scratch110/srpipe/references/Human/default/all/bowtie/human_g1k_v37.fasta.1.ebwt; for i in `seq 8 30`;do bs=$(perl -e "print 2 ** $i");echo -n "$bs";(for x in `seq 1 4`;do ./linear2 $fn &amp; done) | grep Took|tail -1;done<br/> 256 Took 243.905528 seconds<br/> 512 Took 121.146812 seconds<br/> 1024 Took 62.789902 seconds<br/> 2048 Took 30.558433 seconds<br/> 4096 Took 14.908553 seconds<br/> 8192 Took 7.506444 seconds<br/> 16384 Took 3.727981 seconds<br/> 32768 Took 1.864213 seconds<br/> 65536 Took 0.882102 seconds<br/> 131072 Took 0.468359 seconds<br/> 262144 Took 0.405715 seconds<br/> 524288 Took 0.411299 seconds<br/> 1048576 Took 0.420036 seconds<br/> 2097152 Took 0.470832 seconds<br/> 4194304 Took 0.680612 seconds<br/> 8388608 Took 0.776593 seconds<br/> 16777216 Took 0.752007 seconds<br/> 33554432 Took 0.765050 seconds<br/> 67108864 Took 0.741756 seconds<br/> 134217728 Took 0.712213 seconds<br/> 268435456 Took 0.677054 seconds<br/> 536870912 Took 0.753955 seconds<br/> 1073741824 Took 0.931833 seconds</p> <p>I then graphed these. Thin lines are 1 process running. The thicker ones are 4, and the thickest are 32 (also the labels lustre, lustre4 and lustre32 correspond as dot the crosses, circles and squares on the data points).</p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/13883/13883_io.png" style="border: 0px solid black" /></span></p> <p>Naturally for any read there will be a per system-call overhead, so obviously tiny buffers for read calls are inefficient. This is evidently clear for all file systems.</p> <p>Also there is clearly a per-byte overhead. Even reading from a file in cached memory, it takes time to copy from kernel space buffer cache to the user space buffer. This is a per-byte delay and will scale linearly will the data size.</p> <p>These two timings combine to get the net speed of the while(read()) loops. Indeed we see this for NFS. Below 1k block size there is a decrease in speed as the block size decreases, indicating that the overhead of calling more system calls is the major bottleneck. At 10k and above the time is more or less constant, so the per-system call overhead has become insignificant. Inbetween 1k and 10k we see the gradual bend in the graph where real time is more evenly spread between memory copying and per-system call costs. Lustre single-threaded code demonstrates the same bend between 10k and 100k implying the per system-call cost is higher.</p> <p>Now if we have 4 processes running we could expect the per-system call overhead to be the same on all processes, and much the same delay as we get with 1 or 32 processes running (more or less). This is visible in the NFS benchmark as the 32-process version still shows a linear decrease in performance for blocks below 1k (as for the single-process NFS stats) and the bend is "more or less" in the same place.</p> <p>However Lustre shows a marked difference. As we have more and more simultaneous processes it reveals the location of the trade-off between per-system call and per-byte costs moving from ~10K through ~100k for 4 processes and up to ~1Mb for 32 processes. Why is this? Basically it indicates that the CPU overhead of each and every system call goes up as other processes are doing more system calls. The net effect is that even on moderate sized buffers (eg 10K) lustre becomes a couple orders of magnitude slower than NFS.</p> <p>To my eyes that implies there is some contention between independent system calls, possibly a thread lock. I appreciate that coherency is HARD, so I wonder if there is some way we can provide hints to the lustre client to make things easier. Eg is it possible to set the immutable flag on a file to avoid this lock? Does opening in read-only mode help? Does having the file itself be read-only help?</p> <p>James</p> <p>Hi James,</p> <p>Can you confirm what version of Lustre the clients and servers are running? Are you able to get the debug logs that Andreas requested while running these tests? I had to use the debug_daemon function in order to capture all the logs. </p> <p>Thanks,<br/> Kit</p> <p>I have done my test on lustre 1.8.9 clients and 2.5.0 clients however James above would have only used lustre 1.8.9. I am rather deep in a set of lustre server upgrades right now so I will not be able to look at this until Tuesday unless it is critical before then ?</p> <p>James, you are correct that there is per-syscall overhead in Lustre, and contention will indeed increase with multiple threads. Lustre has fully coherent locking between nodes, while NFS has no locking at all. That means if one node is writing to a file and another node is reading at the same time, or if two nodes are writing in on non-block-aligned boundaries, NFS does not guarantee anything about the state of your data at the end. NFS only ensures that local data is flushed when the file is closed, and another client can see this data when the file is opened afterward.</p> <p>That said, I agree the client-side locking overhead is higher than it should be, and we are working to improve this for the 2.6 release. If you are able to test prototype code on a client node, it would be possible to see if the current performance patch series is showing improvement on your system. Most of the patches are already in the lustre-master branch in Git.</p> <p>The last patch in this series that I'm aware of is:<br/> <a href="http://review.whamcloud.com/7895" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7895</a></p> <p>There are pre-built RPMs available at (click on blue circle for your arch distro combo):<br/> <a href="http://build.whamcloud.com/job/lustre-reviews/20005/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/20005/</a></p> <p>This is a set of test results from a production node that has been taken out of one of our clusters, Note that the lustre client is mounted with localflock. We have an expectation that a number of processes all reading the same file should be quicker than a number of processes all reading different files as the file is in cache and we would not expect additional IO however there appears to be significant contention which we do not understand. I note that the block size that we read the file with makes a very significant difference. The problem was discovered with a program that read the file one character at a time. I will repeat the runs with a lustre 2.1 client pulled down from the lustre-reviews.</p> <p>A little table which summarises the results.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre reading multiple files concurrently </td> <td class='confluenceTd'> 4843 secs </td> <td class='confluenceTd'> 7202 secs </td> <td class='confluenceTd'> 1.5 </td> </tr> <tr> <td class='confluenceTd'>lustre reading a single file concurrently </td> <td class='confluenceTd'> 8470 secs </td> <td class='confluenceTd'> 131779 secs </td> <td class='confluenceTd'> 15.6 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading multiple files concurrently </td> <td class='confluenceTd'> 8019 secs </td> <td class='confluenceTd'> 6142 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading a single file concurrently </td> <td class='confluenceTd'> 8161 secs </td> <td class='confluenceTd'> 6131 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> </tbody></table> </div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Lustre 1.8.9 client, lustre 2.4.1 server ( 32 files each 512MB in size , reading with dd 4MB block size), mounted using localflock Creating source files................................ Dropping cache All threads reading the same file Dropping cache Reading the same file in a single thread Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H Pattern takes 4843.25 Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H File_A File_A File_A File_A File_A File_A File_A File_A File_B File_B File_B File_B File_B File_B File_B File_B File_C File_C File_C File_C File_C File_C File_C File_C File_D File_D File_D File_D File_D File_D File_D File_D File_E File_E File_E File_E File_E File_E File_E File_E File_F File_F File_F File_F File_F File_F File_F File_F File_G File_G File_G File_G File_G File_G File_G File_G File_H File_H File_H File_H File_H File_H File_H File_H Pattern takes 8469.83 Lustre 1.8.9 client, lustre 2.4.1 server ( 32 files each 512MB in size , reading with dd 4KB block size), mounted using local flock root@sf-7-1-02:/root# ./test.sh Creating source files................................ Dropping cache All threads reading the same file Dropping cache Reading the same file in a single thread Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H Pattern takes 7202.32 Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H File_A File_A File_A File_A File_A File_A File_A File_A File_B File_B File_B File_B File_B File_B File_B File_B File_C File_C File_C File_C File_C File_C File_C File_C File_D File_D File_D File_D File_D File_D File_D File_D File_E File_E File_E File_E File_E File_E File_E File_E File_F File_F File_F File_F File_F File_F File_F File_F File_G File_G File_G File_G File_G File_G File_G File_G File_H File_H File_H File_H File_H File_H File_H File_H Pattern takes 131779 tmpfs ( 32 files each 512MB in size , reading with dd 4MB block size) Creating source files................................ Dropping cache All threads reading the same file Dropping cache Reading the same file in a single thread Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H Pattern takes 8018.69 Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H File_A File_A File_A File_A File_A File_A File_A File_A File_B File_B File_B File_B File_B File_B File_B File_B File_C File_C File_C File_C File_C File_C File_C File_C File_D File_D File_D File_D File_D File_D File_D File_D File_E File_E File_E File_E File_E File_E File_E File_E File_F File_F File_F File_F File_F File_F File_F File_F File_G File_G File_G File_G File_G File_G File_G File_G File_H File_H File_H File_H File_H File_H File_H File_H Pattern takes 8160.59 tmpfs ( 32 files each 512MB in size , reading with dd 4KB block size) Creating source files................................ Dropping cache All threads reading the same file Dropping cache Reading the same file in a single thread Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H Pattern takes 6141.82 Time taken using read pattern Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H File_A File_A File_A File_A File_A File_A File_A File_A File_B File_B File_B File_B File_B File_B File_B File_B File_C File_C File_C File_C File_C File_C File_C File_C File_D File_D File_D File_D File_D File_D File_D File_D File_E File_E File_E File_E File_E File_E File_E File_E File_F File_F File_F File_F File_F File_F File_F File_F File_G File_G File_G File_G File_G File_G File_G File_G File_H File_H File_H File_H File_H File_H File_H File_H Pattern takes 6130.82 The scripts are as follows: cat readfile.sh #!/bin/sh TYPE=$1 LOC=$2 PAR=$3 MULT=$4 if [ ${TYPE} = "SEQ" ] ; then for i in `seq -w 1 ${PAR}` do echo ${LOC}/${i} for j in `seq 1 ${MULT}` do dd if=${LOC}/${i} of=/dev/null bs=4k done done fi if [ ${TYPE} = "PAR" ] ; then echo $LOC for i in `seq -w 1 ${PAR}` do echo ${LOC} for j in `seq 1 ${MULT}` do dd if=${LOC} of=/dev/null bs=4k done done fi And the test script cat test.sh #!/bin/sh LOC=/mnt/tmp/4k #Where to store the results RES=`echo $LOC| sed -e 's#/#-#g' -e 's/^-//' ` # How many processes to run at one time PAR=32 BASE=`dirname $0` # How many times to read the file ( to multiple the caching effect ) MULT=4 mkdir -p ${RES} echo -n Creating source files for i in `seq -w 1 ${PAR}` do dd if=/dev/zero of=${LOC}/${i} bs=4M count=128 &gt; ${RES}/${i}_create 2&gt;&amp;1 echo -n . done echo echo Dropping cache echo 3 &gt; /proc/sys/vm/drop_caches echo "All threads reading the same file" for i in `seq -w 1 ${PAR}` do (time $BASE/readfile.sh SEQ ${LOC} ${PAR} ${MULT}) &gt; ${RES}/${i}_parallel_same_file 2&gt;&amp;1 &amp; echo -n done wait echo Dropping cache echo 3 &gt; /proc/sys/vm/drop_caches echo "Reading the same file in a single thread " for i in `seq -w 1 ${PAR} ` do (time ${BASE}/readfile.sh PAR ${LOC}/${i} ${PAR} ${MULT} ) &gt; ${RES}/${i}_parallel_different_file 2&gt;&amp;1 &amp; done wait echo "Time taken using read pattern" echo "Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo "FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H" echo -n "Pattern takes " grep -h real ${RES}/*_parallel_different_file | sed -e 's/real *[^ ]//' -e 's/s$//' -e 's/m/*60+/' | bc | awk '{ sum+=$1} END {print sum}' echo "Time taken using read pattern" echo "Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H" echo "File_A File_A File_A File_A File_A File_A File_A File_A" echo "File_B File_B File_B File_B File_B File_B File_B File_B" echo "File_C File_C File_C File_C File_C File_C File_C File_C" echo "File_D File_D File_D File_D File_D File_D File_D File_D" echo "File_E File_E File_E File_E File_E File_E File_E File_E" echo "File_F File_F File_F File_F File_F File_F File_F File_F" echo "File_G File_G File_G File_G File_G File_G File_G File_G" echo "File_H File_H File_H File_H File_H File_H File_H File_H" echo -n "Pattern takes " grep -h real ${RES}/*_parallel_same_file | sed -e 's/real *[^ ]//' -e 's/s$//' -e 's/m/*60+/' | bc | awk '{ sum+=$1} END {print sum}' </pre> </div></div> <p>Hi James,</p> <p>Can you also try with a master branch client? If you have time, a 2.5.x client would also be interesting I think.</p> <p>I'll try to test with different client versions as well. </p> <p>Thanks,<br/> Kit</p> <p>This is from an old system that has only 16Gig of RAM, and only 8 cores. The size of each file was increased to 1G but the amount of parallelism reduced to 8.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 67.821266] Lustre: Lustre: Build Version: jenkins-gb641cc0-PRISTINE-2.6.32-41-server [ 68.182300] Lustre: Added LNI 172.17.96.27@tcp [8/256/0/180] [ 68.182364] Lustre: Accept secure, port 988 [ 68.414016] Lustre: Lustre OSC module (ffffffffa08556e0). [ 68.538800] Lustre: Lustre LOV module (ffffffffa08ea0e0). [ 68.701672] Lustre: Lustre client module (ffffffffa09d6ca0). [ 174.074540] Lustre: MGC172.17.128.135@tcp: Reactivating import [ 174.074569] Lustre: Server MGS version (2.4.1.0) is much newer than client version. Consider upgrading client (2.1.6) [ 174.086076] LustreError: 1200:0:(mgc_request.c:247:do_config_log_add()) failed processing sptlrpc log: -2 [ 174.180021] Lustre: Server lus03-MDT0000_UUID version (2.4.1.0) is much newer than client version. Consider upgrading client (2.1.6) </pre> </div></div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre reading multiple files concurrently </td> <td class='confluenceTd'> 868 secs </td> <td class='confluenceTd'> 1702 secs </td> <td class='confluenceTd'> 2.0 </td> </tr> <tr> <td class='confluenceTd'>lustre reading a single file concurrently </td> <td class='confluenceTd'> 988 secs </td> <td class='confluenceTd'> 4744 secs </td> <td class='confluenceTd'> 4.8 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading multiple files concurrently </td> <td class='confluenceTd'> 424 secs </td> <td class='confluenceTd'> 207 secs </td> <td class='confluenceTd'> 0.5 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading a single file concurrently </td> <td class='confluenceTd'> 537 secs </td> <td class='confluenceTd'> 191 secs </td> <td class='confluenceTd'> 0.4 </td> </tr> </tbody></table> </div> <p>4M read, lustre</p> <p>root@isg-dev4:/root# bash test.sh <br/> Creating source files........<br/> Dropping cache<br/> All threads reading the same file<br/> Dropping cache<br/> Reading the same file in a single thread <br/> pTime taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> Pattern takes 867.87<br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> File_A File_A File_A File_A File_A File_A File_A File_A<br/> File_B File_B File_B File_B File_B File_B File_B File_B<br/> File_C File_C File_C File_C File_C File_C File_C File_C<br/> File_D File_D File_D File_D File_D File_D File_D File_D<br/> File_E File_E File_E File_E File_E File_E File_E File_E<br/> File_F File_F File_F File_F File_F File_F File_F File_F<br/> File_G File_G File_G File_G File_G File_G File_G File_G<br/> File_H File_H File_H File_H File_H File_H File_H File_H<br/> Pattern takes 988.253<br/> root@isg-dev4:/root# </p> <p>4K read,lustre</p> <p>root@isg-dev4:/root# bash test.sh <br/> Creating source files........<br/> Dropping cache<br/> All threads reading the same file<br/> Dropping cache<br/> Reading the same file in a single thread <br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> Pattern takes 1702.89<br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> File_A File_A File_A File_A File_A File_A File_A File_A<br/> File_B File_B File_B File_B File_B File_B File_B File_B<br/> File_C File_C File_C File_C File_C File_C File_C File_C<br/> File_D File_D File_D File_D File_D File_D File_D File_D<br/> File_E File_E File_E File_E File_E File_E File_E File_E<br/> File_F File_F File_F File_F File_F File_F File_F File_F<br/> File_G File_G File_G File_G File_G File_G File_G File_G<br/> File_H File_H File_H File_H File_H File_H File_H File_H<br/> Pattern takes 4744.43<br/> root@isg-dev4:/root# </p> <p>4k reads, tmpfs</p> <p>root@isg-dev4:/root# ./test.sh <br/> Creating source files........<br/> Dropping cache<br/> All threads reading the same file<br/> Dropping cache<br/> Reading the same file in a single thread <br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> Pattern takes 207.198<br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> File_A File_A File_A File_A File_A File_A File_A File_A<br/> File_B File_B File_B File_B File_B File_B File_B File_B<br/> File_C File_C File_C File_C File_C File_C File_C File_C<br/> File_D File_D File_D File_D File_D File_D File_D File_D<br/> File_E File_E File_E File_E File_E File_E File_E File_E<br/> File_F File_F File_F File_F File_F File_F File_F File_F<br/> File_G File_G File_G File_G File_G File_G File_G File_G<br/> File_H File_H File_H File_H File_H File_H File_H File_H<br/> Pattern takes 191.962<br/> root@isg-dev4:/root# </p> <p>4M reads,tmpfs</p> <p>root@isg-dev4:/root# ./test.sh <br/> Creating source files........<br/> Dropping cache<br/> All threads reading the same file<br/> Dropping cache<br/> Reading the same file in a single thread <br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> FIle_A FIle_B FIle_C FIle_D FIle_E FIle_F FIle_G FIle_H<br/> Pattern takes 424.451<br/> Time taken using read pattern<br/> Core_A Core_B Core_C Core_D Core_E Core_F Core_G Core_H<br/> File_A File_A File_A File_A File_A File_A File_A File_A<br/> File_B File_B File_B File_B File_B File_B File_B File_B<br/> File_C File_C File_C File_C File_C File_C File_C File_C<br/> File_D File_D File_D File_D File_D File_D File_D File_D<br/> File_E File_E File_E File_E File_E File_E File_E File_E<br/> File_F File_F File_F File_F File_F File_F File_F File_F<br/> File_G File_G File_G File_G File_G File_G File_G File_G<br/> File_H File_H File_H File_H File_H File_H File_H File_H<br/> Pattern takes 537.192</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre></pre> </div></div> <p>sorry I just saw this ticket today. I didn't read all the comments but the result mentioned at: "James Beal added a comment - 18/Nov/13 4:08 AM" seems quite reasonable to me.</p> <p>For serial reading, only the first read is from the server; the following readings are from client cache so this is why they are super fast.</p> <p>For parallel reading, things are different. Basically only one process issued the reading RPC and the other 19 processes are just waiting for the result of reading. For example, for the first 4M pages, the first process sends read RPC, then other processes try to read the same portion of data but find the pages are locked, so they will have to wait for the RPC to finish; the same thing happened to the 2nd 4M block, and so on.</p> <p>But then wouldn't you expect the first serial process time to be about the same as the first parallel process time? You can see from the data that the first serial process completes in about 1/3 less time than any of the parallel processes. </p> <p>Actually you can't expect that because there must be lot of page lock waiting for parallel processes, but things are much better due to read ahead for serial reading.</p> <p>I note that the files are only being read and the file system is mounted with flocklocal, if we compare the ratio of speeds with tmpfs we see no effective difference with tmpfs compared to a slow down by a factor of 15.6 for lustre. Note we have applications that read files one character at a time. Our current work around involves a hack with a LD_PRELOAD which changes the reads in to memory mapped access..... </p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre reading multiple files concurrently </td> <td class='confluenceTd'> 4843 secs </td> <td class='confluenceTd'> 7202 secs </td> <td class='confluenceTd'> 1.5 </td> </tr> <tr> <td class='confluenceTd'>lustre reading a single file concurrently </td> <td class='confluenceTd'> 8470 secs </td> <td class='confluenceTd'> 131779 secs </td> <td class='confluenceTd'> 15.6 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading multiple files concurrently </td> <td class='confluenceTd'> 8019 secs </td> <td class='confluenceTd'> 6142 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading a single file concurrently </td> <td class='confluenceTd'> 8161 secs </td> <td class='confluenceTd'> 6131 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> </tbody></table> </div> <p>Unless there is a difference in what is being tested between the filesystems, or I'm misunderstanding your results, I find it interesting that Lustre concurrent 4MB reads (@4843s) is significantly <em>faster</em> than tmpfs 4MB reads (@~8019s)? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Even with 4k reads the total time for Lustre (@7202s) is less than the 4MB reads for tmpfs (@80192).</p> <p>It is reasonable that there is overhead for Lustre with smaller read sizes and with concurrent IO compared to tmpfs. Lustre needs to do significantly more work per syscall than tmpfs. Even with a single client, there is locking overhead to maintain consistency (whether it is needed in this particular use or not), overhead in being able to potentially distribute the IO across multiple servers, etc., while tmpfs is as close to a "no-op" as a filesystem can be. That said, I agree the 15x slowdown for 4kB reads is excessive, and indicates there is too much contention/overhead in this part of the code.</p> <p>In 1.8 there was a fast path implemented for small reads, since most of the pages were prefetched via readahead and already in the page cache, so only a minimal amount of work was needed before returning the page to userspace. I think Jinshan is working on a patch to improve this for 2.x as well.</p> <p>Andreas, the numbers I pasted were for a 1.8.9 client..., I will try and see what a 2.5 client gives us.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'> Lustre 1.8.9 </td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre reading multiple files concurrently </td> <td class='confluenceTd'> 4843 secs </td> <td class='confluenceTd'> 7202 secs </td> <td class='confluenceTd'> 1.5 </td> </tr> <tr> <td class='confluenceTd'>lustre reading a single file concurrently </td> <td class='confluenceTd'> 8470 secs </td> <td class='confluenceTd'> 131779 secs </td> <td class='confluenceTd'> 15.6 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading multiple files concurrently </td> <td class='confluenceTd'> 8019 secs </td> <td class='confluenceTd'> 6142 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading a single file concurrently </td> <td class='confluenceTd'> 8161 secs </td> <td class='confluenceTd'> 6131 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> </tbody></table> </div> <p>For all these reads the results should be "in the cache", I will try and do a run with 1 character reads which is what the program which showed this issue in the first place so it is apparent how painful the difference is.</p> <p>The change between lustre 1.8.9 and lustre 2.5.0 looks very unpleasant....., ( The run on one character reads has been running over the weekend and it has not completed the first part yet ).... Given that we are moving to lustre 2.5 as our client on all our hosts very soon I am very wary of these numbers.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre 1.8.9 reading multiple files concurrently </td> <td class='confluenceTd'> 4843 secs </td> <td class='confluenceTd'> 7202 secs </td> <td class='confluenceTd'> 1.5 </td> </tr> <tr> <td class='confluenceTd'>lustre 2.5.0 reading multiple files concurrently </td> <td class='confluenceTd'> 4578 secs </td> <td class='confluenceTd'> 55267 secs </td> <td class='confluenceTd'> 12.1 </td> </tr> <tr> <td class='confluenceTd'>lustre 1.8.9 reading a single file concurrently </td> <td class='confluenceTd'> 8470 secs </td> <td class='confluenceTd'> 131779 secs </td> <td class='confluenceTd'> 15.6 </td> </tr> <tr> <td class='confluenceTd'>lustre 2.5.0 reading a single file concurrently </td> <td class='confluenceTd'> 18500 secs </td> <td class='confluenceTd'> 2033260 secs </td> <td class='confluenceTd'> 110.0 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading multiple files concurrently </td> <td class='confluenceTd'> 8019 secs </td> <td class='confluenceTd'> 6142 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> <tr> <td class='confluenceTd'>tmpfs reading a single file concurrently </td> <td class='confluenceTd'> 8161 secs </td> <td class='confluenceTd'> 6131 secs </td> <td class='confluenceTd'> 0.8 </td> </tr> </tbody></table> </div> <p>It seems bad. Let's start by investigating the single file concurrently read case. Can you please me the output of:</p> <p>lctl get_param llite.*.read_ahead_stats<br/> lctl get_param osc.*.rpc_stats</p> <p>when running 1.8 and 2.5 clients specifically. Please remount the client before running the test case so that we can get accurate stats.</p> <p>Also, can you please run it with single thread on both clients to check how much difference it is between them?</p> <p>To complete the tests in a reasonable time I had to reduce the size of the file to 128MB</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre 1.8.9 reading a single file concurrently </td> <td class='confluenceTd'> 768 secs </td> <td class='confluenceTd'> 5373 secs </td> <td class='confluenceTd'> 7 </td> </tr> <tr> <td class='confluenceTd'>lustre 2.5.0 reading a single file concurrently </td> <td class='confluenceTd'> 1363 secs </td> <td class='confluenceTd'> 132559 secs </td> <td class='confluenceTd'> 97 </td> </tr> <tr> <td class='confluenceTd'> Slow down </td> <td class='confluenceTd'> 1.8 </td> <td class='confluenceTd'> 24.7 </td> <td class='confluenceTd'>&nbsp;</td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre 1.8.9 reading a single file </td> <td class='confluenceTd'> 0.45 secs </td> <td class='confluenceTd'> 0.35 secs </td> <td class='confluenceTd'> 0.78 </td> </tr> <tr> <td class='confluenceTd'>lustre 2.5.0 reading a single file </td> <td class='confluenceTd'> 0.38 secs </td> <td class='confluenceTd'> 1.25 secs </td> <td class='confluenceTd'> 3.29 </td> </tr> <tr> <td class='confluenceTd'> Slow down </td> <td class='confluenceTd'> 0.85 </td> <td class='confluenceTd'> 3.57 </td> <td class='confluenceTd'>&nbsp;</td> </tr> </tbody></table> </div> <p>The single client reads with a larger file</p> <p>This is with a 4G files, which seems to follow the smaller file patten.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K reads </td> <td class='confluenceTd'> slow down </td> </tr> <tr> <td class='confluenceTd'>lustre 1.8.9 reading a single file </td> <td class='confluenceTd'> 9.1 secs </td> <td class='confluenceTd'> 9.0 secs </td> <td class='confluenceTd'> 1.01 </td> </tr> <tr> <td class='confluenceTd'>lustre 2.5.0 reading a single file </td> <td class='confluenceTd'> 8.8 secs </td> <td class='confluenceTd'> 36.9 secs </td> <td class='confluenceTd'> 4.19 </td> </tr> <tr> <td class='confluenceTd'> Slow down </td> <td class='confluenceTd'> 0.97 </td> <td class='confluenceTd'> 4.1 </td> <td class='confluenceTd'>&nbsp;</td> </tr> </tbody></table> </div> <p>The logs have been added as attachments, along with a new copy of the test script.</p> <p>It looks like that per IO overhead of 2.x is much higher than 1.8. I did some experiments on my local machine and cl_lock contributes most of the overhead.</p> <p>I hope we can solve this problem in the on going clio simplification project.</p> <p>Hi Jinshan,</p> <p>Can you elaborate on the clio simplification project and what sort of impact it might have on this IO pattern?</p> <p>Thanks,<br/> Kit</p> <p>Clio project is funded by opensfs. We will simplify cl_lock implementation there. There will be performance tune phase at the end of the project, which will give us a chance to look at this problem.</p> <p>I've ever set up an environment to check what's the major components to slow it down. And from what I have seen, cl_lock() contributes some overhead. Other components for example, cl_io data initialization, refresh layout, etc also contributes significant overhead.</p> <p>I'm on a really tight schedule so I can't find any time to work on this recently.</p> <p>Kit: for clarity, OpenSFS have funded a design phase for CLIO Simplification. Implementation and an associated performance analysis are not within scope for the CLIO Simplification design phase.</p> <p>I'd like to share some result I've got so far.</p> <p>First of all, I can reproduce the problem mentioned by James. So I worked out a patch, plus the prototype of cl_lock simplification, I have some results to share.</p> <p>There are two type of read in my test:<br/> 1. 32 threads read 32 individual files;<br/> 2. 32 threads read the same file.</p> <p>Also run the above test cases with memory in cache and not in cache specifically.</p> <p>Here are the results(seconds):</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> 4M reads </td> <td class='confluenceTd'> 4K read</td> <td class='confluenceTd'> Slow Down</td> </tr> <tr> <td class='confluenceTd'> ind + cache </td> <td class='confluenceTd'> 24.96 </td> <td class='confluenceTd'> 198.53 </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> par + cache </td> <td class='confluenceTd'> 20.65 </td> <td class='confluenceTd'> 111.02 </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> ind + no cache </td> <td class='confluenceTd'> 161.8 </td> <td class='confluenceTd'> 157.37 </td> <td class='confluenceTd'>&nbsp;</td> </tr> <tr> <td class='confluenceTd'> par + no cache </td> <td class='confluenceTd'> 13.44 </td> <td class='confluenceTd'> 115.51 </td> <td class='confluenceTd'>&nbsp;</td> </tr> </tbody></table> </div> <p>ind: threads read individual files<br/> par: threads read the same file<br/> cache: before reading, the data is already in memory<br/> no cache: clean up memory cache before reading</p> <p>Raw data is here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 tests]# CACHE=yes ~/test.sh Dropping cache Read everything into memory cache..................................done Creating source files................................ Block size 4k cache: yes, Threads read individual files - wall: 16, real: 198.53 seconds cache: yes, All threads reading the same file - wall: 43, real: 111.02 seconds Block size 4M cache: yes, Threads read individual files - wall: 1, real: 24.96 seconds cache: yes, All threads reading the same file - wall: 1, real: 20.65 seconds [root@c01 tests]# CACHE=no ~/test.sh Dropping cache Creating source files................................ Block size 4k Dropping cache cache: no, Threads read individual files - wall: 101, real: 157.37 seconds Dropping cache cache: no, All threads reading the same file - wall: 43, real: 115.51 seconds Block size 4M Dropping cache cache: no, Threads read individual files - wall: 94, real: 161.8 seconds Dropping cache cache: no, All threads reading the same file - wall: 1, real: 13.44 seconds </pre> </div></div> <p>and this is my test script:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 tests]# cat ~/test.sh #!/bin/bash CACHE=${CACHE:-"no"} LOC=/mnt/lustre/lustre_test mkdir -p $LOC #Where to store the results RES=~/res # How many processes to run at one time PAR=32 BASE=`dirname $0` # How many times to read the file ( to multiple the caching effect ) MULT=1 rm -rf ${RES} &amp;&amp; mkdir -p ${RES} function remount_lustre { echo Dropping cache echo 3 &gt; /proc/sys/vm/drop_caches umount /mnt/lustre mount | grep c01 mount -t lustre -o localflock c01@tcp0:/lustre /mnt/lustre if [ "$CACHE" = "yes" ]; then echo -n Read everything into memory cache.. for i in `seq -f "%03.f" 1 ${PAR}`; do dd if=${LOC}/${i} of=/dev/null bs=4M &gt; /dev/null 2&gt;&amp;1 echo -n . done echo done fi } remount_lustre echo -n Creating source files for i in `seq -f "%03.f" 1 ${PAR}` do [ -e ${LOC}/${i} ] || dd if=/dev/zero of=${LOC}/${i} bs=4M count=32 &gt; ${RES}/${i}_create 2&gt;&amp;1 echo -n . done echo for BLOCK in "4k" "4M" do echo "Block size $BLOCK" [ "$CACHE" = "no" ] &amp;&amp; remount_lustre echo -n "cache: $CACHE, Threads read individual files - " st=$(date +%s) for i in `seq -f "%03.f" 1 ${PAR}`; do /usr/bin/time -p -a -o ${RES}/single.${BLOCK} dd if=${LOC}/$i of=/dev/null bs=${BLOCK} &gt; /dev/null 2&gt;&amp;1 &amp; done wait echo -n "wall: $((`date +%s`-st)), " t=`grep -h real ${RES}/single.${BLOCK} | sed -e 's/real *[^ ]//' -e 's/s$//' -e 's/m/*60+/' | bc | awk '{ sum+=$1} END {print sum}'` echo "real: $t seconds" echo [ "$CACHE" = "no" ] &amp;&amp; remount_lustre echo -n "cache: $CACHE, All threads reading the same file - " st=$(date +%s) for i in `seq -f "%03.f" 1 ${PAR}`; do /usr/bin/time -p -a -o ${RES}/par.${BLOCK} dd if=${LOC}/001 of=/dev/null bs=${BLOCK} &gt; /dev/null 2&gt;&amp;1 &amp; done wait echo -n "wall: $((`date +%s`-st)), " t=`grep -h real ${RES}/par.${BLOCK} | sed -e 's/real *[^ ]//' -e 's/s$//' -e 's/m/*60+/' | bc | awk '{ sum+=$1} END {print sum}'` echo "real: $t seconds" done [root@c01 tests]# </pre> </div></div> <p>Some of the patches are still in prototype phase, and I'll share the patches with you guys as soon as I can, so that you can try to reproduce what I have done.</p> <p>Today is Chinese New Year. Happy Chinese New Year, everybody!</p> <p>Happy New Year to you to <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I completely understand about prototypes..., It would be good to have before and after numbers in the tables.</p> <p>Thank you for looking at this for us.</p> <p>Could you test with my patch <a href="http://review.whamcloud.com/#/c/9095?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9095?</a><br/> This patch significantly reduce amount of context switches on ll_inode_size_lock(). So, overhead is gone and I have speedup in parallel dds on my machine.</p> <p>I will build a new client tomorrow.</p> <p>Here are the results before and after my patches is applied (in seconds):</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'> single </td> <td class='confluenceTd'> parallel </td> </tr> <tr> <td class='confluenceTd'> master </td> <td class='confluenceTd'> 3.28 </td> <td class='confluenceTd'> 18.91 </td> </tr> <tr> <td class='confluenceTd'> master + patch </td> <td class='confluenceTd'> 2.84 </td> <td class='confluenceTd'> 5.80 </td> </tr> <tr> <td class='confluenceTd'> b1_8 </td> <td class='confluenceTd'> 0.62 </td> <td class='confluenceTd'> 11.54 </td> </tr> </tbody></table> </div> <p>single: single thread read single file<br/> parallel: NR_CPUS threads read single file in parallel(8 threads in my case).</p> <p>the file size is 1G and is already in cache before threads start; the read block size is 4k.</p> <p>While we wait I found the following interesting.</p> <p><a href="http://lwn.net/Articles/590243/" class="external-link" target="_blank" rel="nofollow noopener">http://lwn.net/Articles/590243/</a></p> <p>Performance-oriented patches should, of course, always be accompanied by benchmark results. In this case, Waiman included a set of AIM7 benchmark results with his patch set (which did not include the pending-bit optimization). Some workloads regressed a little, but others shows improvements of 1-2% — a good result for a low-level locking improvement. The disk benchmark runs, however, improved by as much as 116%; that benchmark suffers from especially strong contention for locks in the virtual filesystem layer and ext4 filesystem code.</p> <p>Hi Jinshan,</p> <p>Any updates on the request made by James. We are looking for the new patch for the performance issues to try out, based on the above comment. </p> <p>Thank You,<br/> Manish</p> <p>I came up with a solution for this issue and the initial testing result is exciting.</p> <p>The test case I used is based on the test cases shared by sanger, please check below:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash nr_cpus=$(grep -c ^processor /proc/cpuinfo) <span class="code-keyword">for</span> CACHE in no yes; <span class="code-keyword">do</span> <span class="code-keyword">for</span> BS in 4k 1M; <span class="code-keyword">do</span> echo <span class="code-quote">"===== cache: $CACHE, block size: $BS ====="</span> [ <span class="code-quote">"$CACHE"</span> = <span class="code-quote">"yes"</span> ] &amp;&amp; { dd <span class="code-keyword">if</span>=/mnt/lustre/testfile bs=1M of=/dev/<span class="code-keyword">null</span> &gt; /dev/<span class="code-keyword">null</span> 2&gt;&amp;1; } [ <span class="code-quote">"$CACHE"</span> = <span class="code-quote">"no"</span> ] &amp;&amp; { echo 3 &gt; /proc/sys/vm/drop_caches; } echo -n <span class="code-quote">" single read: "</span> dd <span class="code-keyword">if</span>=/mnt/lustre/testfile bs=$BS of=/dev/<span class="code-keyword">null</span> 2&gt;&amp;1 |grep copied |awk -F, <span class="code-quote">'{print $3}'</span> [ <span class="code-quote">"$CACHE"</span> = <span class="code-quote">"no"</span> ] &amp;&amp; { echo 3 &gt; /proc/sys/vm/drop_caches; } echo -n <span class="code-quote">" parallel read: "</span> <span class="code-keyword">for</span> i in `seq -w 1 ${nr_cpus}`; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=/mnt/lustre/testfile bs=$BS of=/dev/<span class="code-keyword">null</span> &gt; results/${i}_out 2&gt;&amp;1 &amp; done wait grep copied results/1_out | awk -F, <span class="code-quote">'{print $3}'</span> done done </pre> </div></div> <p>The test file is 2G in size so that it can fit in memory for cache enabled testing. I applied the patches and compare the test results w/ and w/o patches.</p> <p>The result w/ my patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>===== cache: no, block size: 4k ===== single read: 1.2 GB/s parallel read: 576 MB/s ===== cache: no, block size: 1M ===== single read: 1.4 GB/s parallel read: 566 MB/s ===== cache: yes, block size: 4k ===== single read: 3.8 GB/s parallel read: 1.8 GB/s ===== cache: yes, block size: 1M ===== single read: 6.4 GB/s parallel read: 1.3 GB/s </pre> </div></div> <p>The test w/o my patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>===== cache: no, block size: 4k ===== single read: 257 MB/s parallel read: 148 MB/s ===== cache: no, block size: 1M ===== single read: 1.1 GB/s parallel read: 420 MB/s ===== cache: yes, block size: 4k ===== single read: 361 MB/s parallel read: 147 MB/s ===== cache: yes, block size: 1M ===== single read: 5.8 GB/s parallel read: 1.3 GB/s </pre> </div></div> <p>The small IO performance improved significantly. I'm still doing some fine tune for the patches and I will release them as soon as I can so that you can do some evaluation.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/20574" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20574</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> clio: replace semaphore with mutex<br/> Project: fs/lustre-release<br/> Branch: b2_4<br/> Current Patch Set: 1<br/> Commit: 3e1cbe0b81eaee6e509c825455669c89df157915</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20254/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20254/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> obdclass: Get rid of cl_env hash table<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 45332712783a4756bf5930d6bd5f697bbc27acdb</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20256/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20256/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> llite: fix up iov_iter implementation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1101120d3258509fa74f952cd8664bfdc17bd97d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20255/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20255/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> llite: fast read implementation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 172048eaefa834e310e6a0fa37e506579f4079df</p> <p>Landed for 2.9</p> <p>This patch introduced an intermittent test failure <a href="https://jira.whamcloud.com/browse/LU-8248" title="sanity test_248: fast read was not 4 times faster" class="issue-link" data-issue-key="LU-8248"><del>LU-8248</del></a> in sanity.sh test_248.</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/20647/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20647/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> test: Correct error_ignore message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5f03bf91e68e925149f2331a44d1e4ad858b8006</p> Duplicate Related LU-8248 LU-9106 LU-7382 LU-9491 LU-4650 LU-4588 Development Rank 1|hzw907: Rank (Obsolete) 11618 Severity