https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-66 Lustre <p>this is just copy of bug 22980, but I think it's better to track &amp; discuss it at here:</p> <p>Hello,</p> <p>Testing our new IO servers we have an issue with obdfilter-survey. Our OSSs are based on 4<br/> Nehalem-EX processors, connected to a Boxboro chipset. Every socket has 6 cores. On every OST we<br/> have several FC channels connected to our storage bay.</p> <p>When we perform raw tests with sgpdd-survey, over 24 luns we get ~4400 MB/s on write and more than<br/> 5500 MB/s on read.</p> <p>Then if we start a Lustre filesystem and we test these 24 osts with obdfilter-survey (size=24192<br/> rszlo=1024 rszhi=1024 nobjlo=1 nobjhi=2 thrlo=1 thrhi=16 case=disk tests_str="write read" sh<br/> obdfilter-survey) we always have a performance limit on 1200 MB/s for write and read.</p> <p>If we perform IOzone tests from five clients (2 threads per client, connected to the server with<br/> Infiniband) we get more than 2500 MB/s.</p> <p>Then we disconnected two sockets using command "echo 0 &gt; /sys/devices/system/cpu/cpu5/online" on<br/> every cpu belonging to these two sockets and we get expected results on obdfilter-survey (4600 MB/s<br/> on write and 5500 MB/s on read). If we only disconnect one socket then obdfilter-survey gives us a<br/> max of 1600 MB/s. Using only one socket results are slightly worse than with two sockets.</p> <p>We also made these tests with Lustre 1.6, with other storage bays and with similar platforms (4<br/> sockets and 8 cpus per socket) having always the same kind of problem. If we activate the<br/> hyper-threading functionality on every socket then performances are even worse. </p> <p>It's like if obdfilter-survey has any kind of saturation when there are many sockets. What do you<br/> think? Thanks,</p> LU-66

obdfilter-survey performance issue on NUMA system

Improvement Minor Resolved Fixed Niu Yawei Liang Zhen Wed, 9 Feb 2011 06:44:03 +0000 Tue, 17 Dec 2013 02:16:38 +0000 Tue, 17 Dec 2013 02:16:38 +0000 Lustre 2.1.0 Lustre 2.1.0 0 8 <p>initial data from Sebastien</p> <p>---------------------------------------</p> <p>Hi,</p> <p>I gave a try to attachment 32668 <span class="error">&#91;details&#93;</span> on Lustre 2.0. I ran the tests on a MESCA server (OSS), running a<br/> RHEL6 kernel (2.6.32).<br/> With this kernel, HAVE_UNLOCKED_IOCTL is defined. But unfortunately, I could see no improvement in<br/> the performance given by obdfilter-survey:</p> <p>Without attachment 32668 <span class="error">&#91;details&#93;</span>, all sockets activated:</p> <p><span class="error">&#91;root@berlin7 ~&#93;</span># numactl -H<br/> available: 4 nodes (0-3)<br/> node 0 cpus: 0 4 8 12 16 20 24 28<br/> node 0 size: 16364 MB<br/> node 0 free: 13521 MB<br/> node 1 cpus: 1 5 9 13 17 21 25 29<br/> node 1 size: 16384 MB<br/> node 1 free: 15637 MB<br/> node 2 cpus: 2 6 10 14 18 22 26 30<br/> node 2 size: 16384 MB<br/> node 2 free: 15584 MB<br/> node 3 cpus: 3 7 11 15 19 23 27 31<br/> node 3 size: 16382 MB<br/> node 3 free: 15977 MB<br/> node distances:<br/> node 0 1 2 3<br/> 0: 10 21 21 21<br/> 1: 21 10 21 21<br/> 2: 21 21 10 21<br/> 3: 21 21 21 10<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span># targets="`lctl dl |grep obdfilter | awk '</p> {print $4}' | tr '\n' ' '`" size=4096<br/> rszlo=1024 rszhi=1024 nobjlo=1 nobjhi=1 thrlo=1 thrhi=128 case=disk tests_str="write read"<br/> rslt_loc=/root/obdsurvey obdfilter-survey<br/> Fri Jan 21 13:37:33 CET 2011 Obdfilter-survey for case=disk from berlin7<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 15 write 611.92 [ 19.00, 54.00] read 1058.04 [ <br/> 48.99, 101.00]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 30 write 1236.58 [ 50.99, 106.98] read 1818.19 [ <br/> 31.98, 367.92]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 60 write 1447.42 [ 10.00, 231.96] read 1928.87 [ <br/> 19.00, 432.96]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 120 write 1632.67 [ 8.00, 341.30] read 1855.03 [ <br/> 0.00, 430.97]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 240 write 1572.07 [ 0.00, 380.62] read 1846.84 [ <br/> 21.00, 385.99]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 480 write 1593.21 [ 11.00, 372.99] read 1811.64 [ <br/> 19.00, 400.68]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 960 write 1508.13 [ 5.00, 380.98] read 1705.11 [ <br/> 3.00, 318.97]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 1920 write 1362.63 [ 1.00, 365.76] read 1595.17 <br/> SHORT<br/> <br/> <br/> <br/> Without attachment 32668 <span class="error">&#91;details&#93;</span>, only one socket activated:<br/> <br/> <span class="error">&#91;root@berlin7 ~&#93;</span># numactl -H<br/> available: 4 nodes (0-3)<br/> node 0 cpus: 0 4 8 12 16 20 24 28<br/> node 0 size: 16364 MB<br/> node 0 free: 13578 MB<br/> node 1 cpus:<br/> node 1 size: 16384 MB<br/> node 1 free: 15862 MB<br/> node 2 cpus:<br/> node 2 size: 16384 MB<br/> node 2 free: 15653 MB<br/> node 3 cpus:<br/> node 3 size: 16382 MB<br/> node 3 free: 16041 MB<br/> node distances:<br/> node 0 1 2 3<br/> 0: 10 21 21 21<br/> 1: 21 10 21 21<br/> 2: 21 21 10 21<br/> 3: 21 21 21 10<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span># targets="`lctl dl |grep obdfilter | awk '{print $4} <p>' | tr '\n' ' '`" size=4096<br/> rszlo=1024 rszhi=1024 nobjlo=1 nobjhi=1 thrlo=1 thrhi=128 case=disk tests_str="write read"<br/> rslt_loc=/root/obdsurvey obdfilter-survey Fri Jan 21 14:10:51 CET 2011 Obdfilter-survey for<br/> case=disk from berlin7<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 15 write 618.98 [ 25.00, 54.99] read 2725.46 [<br/> 121.00, 370.30]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 30 write 1328.17 [ 62.99, 118.98] read 3139.51 [<br/> 106.98, 453.98]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 60 write 1895.78 [ 63.00, 240.98] read 3193.16 [ <br/> 78.93, 434.98]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 120 write 2579.81 [ 36.00, 374.94] read 2845.06 [ <br/> 76.00, 509.88]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 240 write 2177.08 [ 54.00, 386.80] read 2924.08 [ <br/> 44.00, 438.97]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 480 write 1939.15 [ 6.00, 360.85] read 2506.11 [ <br/> 17.98, 363.98]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 960 write 2106.54 [ 2.00, 332.44] read 2378.08 [ <br/> 65.00, 417.72]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 1920 write 1545.92 [ 1.00, 386.49] read 2059.07 <br/> SHORT</p> <p>With attachment 32668 <span class="error">&#91;details&#93;</span>, all sockets activated:</p> <p><span class="error">&#91;root@berlin7 ~&#93;</span># numactl -H<br/> available: 4 nodes (0-3)<br/> node 0 cpus: 0 4 8 12 16 20 24 28<br/> node 0 size: 16364 MB<br/> node 0 free: 13507 MB<br/> node 1 cpus: 1 5 9 13 17 21 25 29<br/> node 1 size: 16384 MB<br/> node 1 free: 15534 MB<br/> node 2 cpus: 2 6 10 14 18 22 26 30<br/> node 2 size: 16384 MB<br/> node 2 free: 15558 MB<br/> node 3 cpus: 3 7 11 15 19 23 27 31<br/> node 3 size: 16382 MB<br/> node 3 free: 15911 MB<br/> node distances:<br/> node 0 1 2 3<br/> 0: 10 21 21 21<br/> 1: 21 10 21 21<br/> 2: 21 21 10 21<br/> 3: 21 21 21 10<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span>#<br/> <span class="error">&#91;root@berlin7 ~&#93;</span># targets="`lctl dl |grep obdfilter | awk '</p> {print $4} <p>' | tr '\n' ' '`" size=4096<br/> rszlo=1024 rszhi=1024 nobjlo=1 nobjhi=1 thrlo=1 thrhi=128 case=disk tests_str="write read"<br/> rslt_loc=/root/obdsurvey obdfilter-survey<br/> Fri Jan 21 14:39:14 CET 2011 Obdfilter-survey for case=disk from berlin7<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 15 write 651.19 [ 27.99, 52.99] read 2622.92 [ <br/> 61.99,1419.80]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 30 write 1249.90 [ 51.99, 112.97] read 2185.82 [ <br/> 37.98, 766.90]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 60 write 1569.65 [ 55.98, 204.97] read 2069.03 [ <br/> 16.97, 610.65]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 120 write 1648.17 [ 28.90, 345.93] read 2011.21 [ <br/> 43.30, 788.25]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 240 write 1551.18 [ 14.73, 395.89] read 1987.69 [ <br/> 60.77, 520.36]<br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 480 write 1582.04 [ 9.29, 338.90] read 1943.12 [ <br/> 58.36, 570.59] <br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 960 write 1480.17 [ 6.00, 292.76] read 1899.71 [ <br/> 60.11, 444.05] <br/> ost 15 sz 62914560K rsz 1024K obj 15 thr 1920 write 1277.28 [ 2.98, 312.91] read 1844.78 [ <br/> 50.56, 432.15] </p> <p>So it seems this lock contention is not the most limiting factor here.</p> <p>HTH,<br/> Sebastien.</p> <p>This is already being discussed in bug <a href="https://jira.whamcloud.com/browse/LU-29" title="obdfilter-survey doesn&#39;t work well if cpu_cores (/w hyperT) &gt; 16" class="issue-link" data-issue-key="LU-29"><del>LU-29</del></a>, so I don't think we need this bug.</p> <p>Andreas, this ticket is created for tracking efforts for Bull, also, this one is about NUMA performance and <a href="https://jira.whamcloud.com/browse/LU-29" title="obdfilter-survey doesn&#39;t work well if cpu_cores (/w hyperT) &gt; 16" class="issue-link" data-issue-key="LU-29"><del>LU-29</del></a> is more about SMP scalability (BKL).</p> <p>Yes, and <a href="https://jira.whamcloud.com/browse/LU-29" title="obdfilter-survey doesn&#39;t work well if cpu_cores (/w hyperT) &gt; 16" class="issue-link" data-issue-key="LU-29"><del>LU-29</del></a> is 1.8.6 blocker, and Peter Jones said NUMA support isn't 1.8.6 goal, so we'd better open another ticket for it.</p> <p>Copy form b22980:</p> <p>Thank you for your review, Andreas. As we discussed in the Jira system, I think this issue might be<br/> NUMA dependent, because the 'unlocked_ioctl' patch works for another customer who runs test over<br/> SMP (though he just compared 24 cores).</p> <p>So next setp, I want Sebanstien to run some tests:</p> <ul class="alternate" type="square"> <li>We just compared 32 cores with patch applied, I think we'd better get the numbers for 1,2,3,4<br/> sockets enabled (8,16,24,32 cores) with/without patch applied(unlocked_ioctl patch).</li> <li>To verify if it's NUMA dependent issue, we also want the numbers for 8,16,24,32 cores enabled,<br/> but this time the enabled cores are distributed on different sockets evenly. (enable 2,4,6,8 cores<br/> on each socket)</li> </ul> <p>At the same time, we want collect some oprofile data during above tests.</p> <p>BTW: I'm wondering how Iozne get the 2500M/s (mentioned in comment #1), how many objects in the<br/> Iozne test? and how the number is measured?</p> <p>Hi,</p> <p>Concerning IOZone, comment #1 in bug 22980 dates back from June 2010, so I do not have the full details about this.<br/> In this case, we ran IOzone with approximately the following command lines, in parallel on 5 clients:</p> <p>Write: iozone -s 4g -r 1024 -+n -c -e -i 0 -t 2 -F /lustre/dir&lt;X&gt;/file1 /lustre/dir&lt;X&gt;/file2<br/> Read: iozone -s 4g -r 1024 -+n -c -e -i 1 -t 2 -F /lustre/dir&lt;X&gt;/file1 /lustre/dir&lt;X&gt;/file2</p> <p>I do not remember the stripe count, probably 2 or 4.<br/> The IOZone figures we gave are the ones directly returned by IOZone ('Children see throughput').</p> <p>I will be able to run the obdfilter-survey tests you are asking for, but I would really appreciate if you could you specify the exact and precise oprofile command lines for the stats you want.</p> <p>TIA,<br/> Sebastien.</p> <p>Hi, Sebastien</p> <p>I see, I was just wondering if the object count of iozone test is same with obdfilter-survey test, and how the aggreated throughtput is calculated. Anyway, let's focus on obdfilter-survey test at this moment.</p> <p>For the oprofile command, I think just general usage is fine:</p> <ul class="alternate" type="square"> <li>opcontrol --vmlinux=/patch/to/vmlinux</li> <li>opcontrol --start</li> <li>run test</li> <li>opcontrol --dump</li> <li>opreport -l (the output is what we want)</li> <li>opcontrol --reset</li> </ul> <p>I'm not oprofile expert, if you meet any trouble with running oprofile, you can ask Liang for help, I believe he played oprofile a lot when he was working on SMP performance.</p> <p>Sebastien,</p> <p>Could you also collect numastat at begin/end of each test (or you know any better way to collect stats for NUMA behavor)? so we can understand better about whether it's just about cross-node data-traffic or not</p> <p>Thanks<br/> Liang</p> <p>obdfilter-survey results, without oprofile and numastat.</p> <p>Sebastien,<br/> thanks for you data.<br/> I have a couple of questions about your server (learnt from your presentation on LUG...):</p> <ul class="alternate" type="square"> <li>I assume you have 2 IOH on the server right?</li> <li>if you have IOHs &gt; 1, how OSTs are distributed on IOHs? 1,3,5,7... on IOH1, and 0,2,4,6... on IOH2?</li> </ul> <p>Thanks<br/> Liang</p> <p>Hi,</p> <p>Yes you are right, we have 2 IOHs on the OSS. Half of the OSSes are directly connected to the first IOH, and the other half is directly connected to the second IOH.</p> <p>Sebastien.</p> <p>Sebastien, so I think each NUMIOA node contains 2 sockets (or numa nodes) right? could you give us detail information about how CPU nodes are distributed on NUMIOA node? i.e:<br/> numioa node0 == socket0 + socket1 + IOH0, numioa node1 == socket2 + socket3 + IOH1 or<br/> numioa node0 == socket0 + socket2 + IOH0, numioa node1 == socket1 + socket3 + IOH1 or<br/> something like that?</p> <p>We are working on two directions right now:</p> <ul class="alternate" type="square"> <li>try to find out whether there are more hight contention locks like BKL, oprofile can help this</li> <li>make lctl processes to be numioa node affinity, Niu is working on a prototype patch</li> </ul> <p>Thanks<br/> Liang</p> <p>Yes, our MESCA machine is made of 2 NUMIOA nodes:<br/> numioa node 0 == socket0 + socket1 + IOH0<br/> numioa node 1 == socket2 + socket3 + IOH1</p> <p>Cheers,<br/> Sebastien.</p> <p>patch to set cpu affinity for lctl test_brw processes and an example config file. </p> <p>Hi, Sebastien</p> <p>Thanks for your test result, looks the patch improves write performance little when there are limited cores (1,2 sockets, 8,16 cores), for 24,32 cores, I don't see improvements. What surprised me is that the read performance has a big improvment when cores are distributed on different sockets, I don't know the reason so far.</p> <p>To testify if the issue is NUMIOA dependent (degradation caused by accessing remote memory/IOH), I made a patch to set cpu affinity for lctl brw_test threads, we want use this patch to collect more data for further analysis. (the patch and the example config file are attached, the config file pathname should be "/tmp/affinity_map") </p> <p>Not sure if we can access your machine now (I think Liang has provided a static address 99.96.190.234), if it's already acceesable for us, pelease send us a guide on how to run tests on your machine as well, then we can run tests by ourselves from now on. Thank you.</p> <p>BTW: could you also provide your sgpdd_survey command which mentioned in the first comment?</p> <p>Full obdfilter-survey results. In the tarball please find:</p> <ul class="alternate" type="square"> <li>summary.txt: array that sums up test results</li> <li>result_*.txt: results for a specific test, with also 'numastat' output</li> <li>opreport_*.txt: associated oprofile data</li> </ul> <p>As you can see, I was not able to run the tests in the '2 sockets' configuration while collecting data with oprofile. Node was crashing in the middle of obdfilter-survey, and I do not know who to blame here...</p> <p>Thanks a lot, Sebastien. What's the kernel version did you run the test on?</p> <p>We use a custom kernel, based on RHEL6 GA (2.6.32-71.el6.x86_64).</p> <p>The test system we are dedicating to you is not ready yet, so I will have to run the tests by myself.</p> <p>I will try lctl_setaffinity.patch, but could you please tell me what kind of tests do you need? Only in 'thread' mode right? Still with oprofile and numastat? How many cores/sockets activated? With or without 'unlocked_ioctl' patch?</p> <p>TIA,<br/> Sebastien.</p> <p>Hi, Sebastien</p> <p>I'd like you to run two tests, one in 'thread' mode, and another in 'objid' mode:</p> <ul class="alternate" type="square"> <li>with 'unlocked_ioctl' patch;</li> <li>24 cores actived; (cores distributed on 4 sockets);</li> <li>with oprofile and numastats;</li> <li>provide the /tmp/obdfilter_survey_xxxx.detail as well, where the thread/object cpu mapping is logged.</li> </ul> <p>In the 'objid' mode, you have to provide objid to cpu core map in the config file, so you should know the object ids and try to mapp them to appropriate cpus (to make cpu always access the local IOH) before the test. Thanks.</p> <p>change vmalloc() to kamlloc() in the iocl path.</p> <p>Hi, Sebastien</p> <p>The oprfile data provided by you is very helpful, in the unpatched tests, we can see thread_return() has extremly high rank, I think it's caused by the contention on BKL; in the patched (with unlocked_ioctl) tests, we can see alloc_vmap_area() and find_vmap_area() have very high rank, I think it's caused by the contention on the vmap_area_lock.</p> <p>I made a patch (remove_vmalloc.patch) which change the vmalloc() to kmalloc() in ioctl path, which could eliminate the contention on vmap_area_lock. Before you run the tests which I suggested in my last comment, I really like you to run this patch (togehter with the unlock_ioctl patch) first to see what'll happen. (of course, please enable oprofile while running tests). Thank you.</p> <p>Change the vmalloc to kmalloc in ioctl path. (the previous one isn't correct, updated with this one)</p> <p>Full obdfilter-survey results with unlocked_ioctl and remove_vmalloc patches. In the tarball please find:</p> <ul> <li>summary.txt: array that sums up test results</li> <li>result_*.txt: results for a specific test, with also 'numastat' output</li> <li>opreport_*.txt: associated oprofile data</li> </ul> <p>Thanks for your testing, Sebastien.</p> <p>The result shows both read and write performance got huge improvement, and the oprofile data looks normal this time. So I think the degradation is caused by the contention on BKL and vmap_area_lock. What I don't see is why the read throughput is extreme high in some cases (more than 10000 MB/s), what's the raw bandwith of each OST?</p> <p>You're welcome.</p> <p>The storage array we are attached to should not give us more than 5 GB/s (read and write). So I think the figures given by obdfilter-survey are inaccurate because the test is not longer enough. Maybe I should increase size.</p> <p>Do you still need me to run affinity tests?</p> <p>Cheers,<br/> Sebastien.</p> <p>I don't think we need run the affinity tests, thank you.</p> <p>I agree that we don't need to run affinity tests because numastat shows that foreign memory access is not a big issue (&lt; 5%). However, I do think that we should increase size (probably 5X) so we can get better vision.<br/> Sebastien, could you please help us to run:</p> <ul class="alternate" type="square"> <li>increase size (5X)</li> <li>only run with patches (kmalloc patch and remove BKL patch)</li> <li>only run with 1,2,3,4 sockets (don't need to iterate over 8,16,24 cores)</li> <li>if it's possible, could you give us sgp-dd results on the same hardware, so we can see whether there is anything else we can improve.</li> </ul> <p>Thanks<br/> Liang</p> <p>New results with unlocked_ioctl and remove_vmalloc patches. In the tarball please find:</p> <ul> <li>result_*.txt: results for a specific obdfilter-survey test, with also 'numastat' output</li> <li>opreport_*.txt: associated oprofile data</li> <li>sgpdd_res.txt : sgpdd_survey results</li> </ul> <p>I am sorry, I was not able to get results for 3, 2 and 1 socket. I launched the tests several times, and each time the server crashed. I seems the system does not appreciate to run oprofile with not all sockets...</p> <p>sgpdd_survey results clearly show a limit around 3 GB/s. This limitation is due to the available bandwidth to the storage, because we use only 4 FC links.</p> <p>Hi, Sebastien</p> <p>The results looks really good, I think it's basically what we expected, thank you.</p> <p>One thing unknown is that the write performance dropped a lot in 960 threads. To measure how the cpu affinity affect test result, could you help us to do more tests? I think it'll be useful for our further performance tuning work.</p> <p>What I want to test is:</p> <ul class="alternate" type="square"> <li>apply "remove BKL" + "kmalloc" + "lctl_setaffinity" patches;</li> <li>run test in "objid" mode, 4 sockets enabled, and without oprofile enabled.</li> <li>provide the result, numstat and the /tmp/obdfilter_survey_xxxx.detail (where the thread/object cpu mapping is logged)</li> </ul> <p>In the "objid" mode, each lctl thread will be mapped to a specified cpu, so you should know all the objids before run tests and set the objid-cpu mapping in the /tmp/affinity_map (please refer to the affinity_map example), of course, the objid should be on the local IOH of it's mapped cpu.</p> <p>If I understand correctly, in order to know in advance the objids I should have a look at the obdfilter_survey_xxxx.detail file and consider the next run will do '+1' on the ids.<br/> The problem is the obdfilter_survey_xxxx.detail contains the following:</p> <p>=======================&gt; ost 15 sz 314572800K rsz 1024K obj 15 thr 15<br/> =============&gt; Create 1 on localhost:quartcel-OST0005_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0008_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0007_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST000c_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0000_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST000d_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0004_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0006_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0002_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST000a_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0009_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST000b_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST000e_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0001_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29<br/> =============&gt; Create 1 on localhost:quartcel-OST0003_ecc<br/> create: 1 objects<br/> create: #1 is object id 0x29</p> <p>So, as you can see, all objects have the same ids on the OSTs... In that case, I am afraid the 'objid to core' mapping is useless.<br/> Unless I manually create new objects on OSTs so that objids are different everywhere?</p> <p>Sebastien.</p> <p>Hi, Sebastien</p> <p>Ah, right. Binding object id to cpu doesn't make sense for this test, I've changed the patch to bind devno to cpu (lctl_setaffinity_v2.patch), and I also updated the example affinity_map. Please use the new patch to run the test that I meantioned in my previous comment. Thanks for your effort!</p> <p>Results with unlocked_ioctl and remove_vmalloc patches, plus affinity patch for obdfilter-survey. In the tarball please find:</p> <ul> <li>summary.txt: array that sums up test results</li> <li>result_*.txt: results for a specific obdfilter-survey test, with also 'numastat' output</li> <li>obdfilter_survey*.detail: associated obdfilter-survey detailed data</li> <li>affinity_map.*: associated affinity mapping</li> <li>sgpdd_res_new.txt : sgpdd_survey results</li> </ul> <p>I am sorry to post these results today, but Jira was not accessible yesterday afternoon (French time). I took the opportunity of running some more tests, with on-purpose bad affinities.<br/> I also ran sgpdd-survey tests, because the hardware on which these tests were launched is not the same as before. Due to software reconfiguration, we now have access to 16 LUNs (instead of 15) through 8 FC links (instead of 4). So raw performance is better than before, as we are not more limited by FC bandwidth.</p> <p>In the results, several points are surprising:</p> <ul class="alternate" type="square"> <li>obdfilter-survey results are better than sgpdd-survey results in write;</li> <li>affinity mapping has little impact on performance, good mapping is better than bad mapping only with a high number of threads.</li> </ul> <p>Hi, Sebastien</p> <p>I agree with your conclusion that NUMIOA affinity doesn't affect test result much, to double confirm it, I think we should run the obdfilter-survey (without affinity patch) once more on the new system to see if there is any difference.</p> <p>For the issue of sgpdd-survey write worse than obdfilter-survey, I think one possible reason is that sgpdd-survey used few threads, however, the result shows 128 threads number is very close to 64 threads', maybe there is some bottleneck in the sgpdd-survey test tool, I will look into the code to get more details. At the same time, I think two quick tests could be helpful for this investigation:</p> <ul class="alternate" type="square"> <li>run sgpdd-survey on only one device, try to get the raw bandwith of each single device;</li> <li>collect oprofile data while running sgpdd-survey over 16 devices to see if there is any contention;</li> </ul> <p>I've got the access to the test system today, but it'll take me some time to learn how to run these tests on it, could you help me to run above three tests this time? Thanks a lot.</p> <p>I ran bunch of sgpdd-survey tests on berlin6, and the oprofile result shows more than 50% copy_user_generic_string() samples. Since sgpdd-survey calls read/write syscall to perform I/O against block device, there should be lots of copy_from/to_user() for transfering data between userspace and kernel, and that consumes lots of CPU time. However, I don't think the copy_from_user() is the major bottoleneck of sgpdd-survey write, because when I ran sgpdd-survey read only, the copy_user_generic_string() samples is still very high (more than 60%), but sgpdd-survey read performance is comparable with obdfilter-survey's.</p> <p>The sgp_dd calls it's own sg_write()/sg_read(), and sg_read/write() just simply generate io request for underlying device, then unlug the device (that explains why iostat doesn't work for sgp_dd tests, because it bypass kernel io statistic code). I think tbe bottleneck should be in sg_write() (maybe it doesn't work well in multi-cores/mult-threads condition), though the exact root cause is unkown yet.</p> <p>performance charts for obdfilter-survey &amp; sgpdd-survey<br/> btw, non-patched data is not here because it's too difficult for us to reinstalled unpatched rpms remotely, so we don't have data for unpatched results on the same machine.</p> <p>sorry, some data in previous file is wrong</p> <p>performance graphs (adding data for non-patched version)</p> <p>latest testing results graphs</p> <p>I think this can be closed now.</p> Bugzilla ID 22980.0 Development Rank 1|hzvsmf: Rank (Obsolete) 8541