Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-276] ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %) https://jira.whamcloud.com/browse/LU-276 Lustre <p>We've seen this bug again it's supposed to be fixed in bug 23408. Need to figure out why it became wrong again.</p> <p>Also, there is alignment problem in the code of ost-pools:test-18, needs to be fixed as well.</p> LU-276

ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)

Bug Major Resolved Incomplete Hongchao Zhang Jinshan Xiong Wed, 4 May 2011 11:51:18 +0000 Thu, 7 Jan 2016 23:40:53 +0000 Thu, 7 Jan 2016 23:40:53 +0000 Lustre 2.1.0 Lustre 2.2.0 Lustre 2.1.1 0 6 <p>FWIW, Wangdi landed a patch to master to fix the problem of qos_remedy_create() which could allocate objects outside the pool.<br/> That was bug 21379 & commit eb7c28ff977f4e0a280558aa74e23f2a9ab0ea0c.<br/> This patch adds a call to lov_find_pool() in qos_remedy_create() which is going to print another CWARN(). Maybe that's the reason why we see this failure now and not before (and not on b1_8 where this patch is not landed). </p> <p>HongChao</p> <p>This is causing failures with some of the automated test runs so could you please look into this as a priority</p> <p>Thanks</p> <p>Peter</p> <p>take it and has started to work on it.</p> <p>there are several cases leading to performance degradation while using pool,<br/> 1, the pool lookup in hash table<br/> 2, the extra "CWARN" debug info if the specified pool doesn't exist</p> <p>but local tests show no big difference in these 3 case(without pool, wide pool, missing pool),<br/> the difference tends to decrease if the created files increase, and will be same if the files count is larger than 20000</p> <p>the current file number 9877 is a little small, how about increasing it to a bigger value to lessen the affect of those<br/> random condition which could lead to false result?</p> <p>the tests in Toro show the same result, will create a patch to increase the count of <br/> the created files in the test_18 to see whether the issue can be fixed</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=ofa/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,ofa #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/121/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #121</a><br/> <a href="https://jira.whamcloud.com/browse/LU-276" title="ost-pools: test-18 Degradation with missing pool is 26.07 % (> 15 %)" class="issue-link" data-issue-key="LU-276"><del>LU-276</del></a> increase file count to mitigate affect of random condition</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=016c5a0f6e7307a2a3e05eafa8a36ac16b209643" class="external-link" target="_blank" rel="nofollow noopener">016c5a0f6e7307a2a3e05eafa8a36ac16b209643</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Fix landed for 2.1. Please reopen if issue reoccurs or more work is still required</p> <p>Lustre Clients:<br/> Tag: 1.8.6-wc1<br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32_131.2.1.el6)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/</a><br/> Network: IB (inkernel OFED)<br/> ENABLE_QUOTA=yes</p> <p>Lustre Servers:<br/> Tag: v2_0_66_0<br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.2.1.el6_lustre)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/228/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-master/228/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/</a><br/> Network: IB (inkernel OFED)</p> <p>ost-pools test 18 failed with the similar issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Avg time taken for 9877 creates without pool: 16.32 Avg time taken for 9877 creates with pool: 19.20 Avg time taken for 9877 creates with missing pool: 19.40 No pool to wide pool: 17.64 %. ost-pools test_18: @@@@@@ IGNORE (bz23408): Degradation with wide pool is 17.64 % (> 15 %) </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/02ffc662-b91b-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/02ffc662-b91b-11e0-8bdf-52540025f9af</a></p> <p>the files is still 9877 files in the new occurrence for the patch was only landed on master, but the test is run on b1_8<br/> yujian will try to re-test it with 30000 files soon.</p> <p>With "numfiles=30000", the test 18 still failed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== test 18: File create in a directory which references a deleted pool == 00:50:15 Create performance, iteration 1, 30000 files x 3 total: 30000 creates in 50.11 seconds: 598.69 creates/second iter 1: 30000 creates without pool: 50.11 fat-amd-1-ib: Pool lustre.pool1 created fat-amd-1-ib: OST lustre-OST0000_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID added to pool lustre.pool1 total: 30000 creates in 68.71 seconds: 436.59 creates/second iter 1: 30000 creates with pool: 68.71 fat-amd-1-ib: OST lustre-OST0000_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID removed from pool lustre.pool1 fat-amd-1-ib: Pool lustre.pool1 destroyed total: 30000 creates in 64.46 seconds: 465.39 creates/second iter 1: 30000 creates with missing pool: 64.46 Create performance, iteration 2, 30000 files x 3 total: 30000 creates in 49.84 seconds: 601.93 creates/second iter 2: 30000 creates without pool: 49.84 fat-amd-1-ib: Pool lustre.pool1 created fat-amd-1-ib: OST lustre-OST0000_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID added to pool lustre.pool1 total: 30000 creates in 61.67 seconds: 486.42 creates/second iter 2: 30000 creates with pool: 61.67 fat-amd-1-ib: OST lustre-OST0000_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID removed from pool lustre.pool1 fat-amd-1-ib: Pool lustre.pool1 destroyed total: 30000 creates in 69.00 seconds: 434.80 creates/second iter 2: 30000 creates with missing pool: 69.00 Create performance, iteration 3, 30000 files x 3 total: 30000 creates in 50.25 seconds: 597.06 creates/second iter 3: 30000 creates without pool: 50.25 fat-amd-1-ib: Pool lustre.pool1 created fat-amd-1-ib: OST lustre-OST0000_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID added to pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID added to pool lustre.pool1 total: 30000 creates in 69.04 seconds: 434.50 creates/second iter 3: 30000 creates with pool: 69.04 fat-amd-1-ib: OST lustre-OST0000_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0001_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0002_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0003_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0004_UUID removed from pool lustre.pool1 fat-amd-1-ib: OST lustre-OST0005_UUID removed from pool lustre.pool1 fat-amd-1-ib: Pool lustre.pool1 destroyed total: 30000 creates in 63.49 seconds: 472.49 creates/second iter 3: 30000 creates with missing pool: 63.49 Avg time taken for 30000 creates without pool: 50.06 Avg time taken for 30000 creates with pool: 66.47 Avg time taken for 30000 creates with missing pool: 65.65 No pool to wide pool: 32.78 %. ost-pools test_18: @@@@@@ IGNORE (bz23408): Degradation with wide pool is 32.78 % (> 15 %) Dumping lctl log to /home/yujian/test_logs/2011-07-29/004914/ost-pools.test_18.*.1311927141.log No pool to missing pool: 31.14 %. ost-pools test_18: @@@@@@ IGNORE (bz23408): Degradation with missing pool is 31.14 % (> 30 %) Dumping lctl log to /home/yujian/test_logs/2011-07-29/004914/ost-pools.test_18.*.1311927144.log Resetting fail_loc on all nodes...done. </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/3f75d49c-b9bb-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3f75d49c-b9bb-11e0-8bdf-52540025f9af</a></p> <p>it's related to 1.8<<del>>2.1, the creation performance is almost the same for 2.1<</del>>2.1, but show much difference for 1.8<->2.1</p> <p>So, is the action here to land this fix on the 1.8.x branch</p> <blockquote><p>So, is the action here to land this fix on the 1.8.x branch</p></blockquote> <p>Unfortunately, the fix does not resolve the issue under the 1.8<->2.1 interop configuration. I think Hongchao is still investigating.</p> <p>Hi Hongchao,<br/> Per your suggestion, I ran the test on the latest b1_8 clients with the latest master servers, the issue still existed:<br/> <a href="https://maloo.whamcloud.com/test_sets/869c0eea-be6b-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/869c0eea-be6b-11e0-8bdf-52540025f9af</a></p> <p>Here is the configuration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre Clients: Branch: b1_8 Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32_131.2.1.el6) Build: http://newbuild.whamcloud.com/job/lustre-b1_8/119/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/ Lustre Servers: Branch: master Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6_lustre) Build: http://newbuild.whamcloud.com/job/lustre-master/240/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/ Network: IB (inkernel OFED) ENABLE_QUOTA=yes </pre> </div></div> <p>Lustre Clients:<br/> Tag: 1.8.6-wc1<br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32_131.2.1.el6)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/</a><br/> Network: IB (inkernel OFED)<br/> ENABLE_QUOTA=yes</p> <p>Lustre Servers:<br/> Tag: v2_1_0_0_RC1<br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6_lustre)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/271/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-master/271/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/</a><br/> Network: IB (inkernel OFED)</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/9207414a-cf7e-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9207414a-cf7e-11e0-8d02-52540025f9af</a></p> <p>Hit the same issue when running interop test between 1.8.7-wc1 and 2.1.55<br/> <a href="https://maloo.whamcloud.com/test_sets/fca92dd6-4b0e-11e1-915b-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fca92dd6-4b0e-11e1-915b-5254004bbbd3</a></p> <p>I believe Andreas is fixing this under <a href="https://jira.whamcloud.com/browse/LU-1042" title="1.8 clients show wrong dates with 2.1 servers" class="issue-link" data-issue-key="LU-1042"><del>LU-1042</del></a></p> <p>this issue could be related the config reprocess due to the config change caused by the pool's operation,<br/> it's better to skip the first creation operations in the "with-pool" case and "with-missing-pool" case, will submit<br/> a patch to verify whether it's the case.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,2136" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2136</a></p> <p>Lustre Clients:<br/> Tag: 1.8.7-wc1<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-274.3.1.el5)<br/> Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/171/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/171/</a><br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>Lustre Servers:Tag: v2_1_1_0_RC2<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-274.12.1.el5_lustre.g4554b65)<br/> Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/41/</a><br/> Network: TCP (1GigE)</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/32ec951e-587e-11e1-a226-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/32ec951e-587e-11e1-a226-5254004bbbd3</a></p> <p>Marking this as resolved/incomplete, given the length of time since it was last updated.</p> <p>If anyone disagrees, or would like this ticket re-opened, please let us know.<br/> Thanks,<br/> ~ jfc.</p> Bugzilla ID 23408.0 Development Rank 1|hzv99r: Rank (Obsolete) 5036 Severity