Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-797] Test failure on test suite ost-pools, subtest test_14, test_18, test_23 https://jira.whamcloud.com/browse/LU-797 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/f6c809ee-00ec-11e1-bb4f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f6c809ee-00ec-11e1-bb4f-52540025f9af</a>.</p> <p>The sub-test test_14 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: ost-pools 14</p> LU-797

Test failure on test suite ost-pools, subtest test_14, test_18, test_23

Bug Blocker Resolved Fixed Niu Yawei Maloo Thu, 27 Oct 2011 19:33:00 +0000 Fri, 22 Feb 2013 13:24:14 +0000 Wed, 19 Dec 2012 11:39:03 +0000 Lustre 2.2.0 Lustre 2.1.2 Lustre 2.1.3 Lustre 2.1.4 Lustre 1.8.8 Lustre 2.4.0 Lustre 2.1.4 Lustre 1.8.9 0 8 <p>Niu will look into this one</p> <p>Look at the client test log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Filling up OST0 14188+0 records in 14188+0 records out 14877196288 bytes (15 GB) copied, 836.591 seconds, 17.8 MB/s 14188+0 records in 14188+0 records out 14877196288 bytes (15 GB) copied, 908.419 seconds, 16.4 MB/s 14188+0 records in 14188+0 records out 14877196288 bytes (15 GB) copied, 922.504 seconds, 16.1 MB/s </pre> </div></div> <p>It's filling up OST0 (which is ~150G), and it has already used (836 + 908 + 922)=2666 seconds for only 45G, however the timeout of this test is 3600, so it just doesn't have enough time to fill up the OST0.</p> <p>I think this is a test issue, should we adjust the timeout according to the OST size or just skip the test if the OST is too big?</p> <p>This is similar to test 220 issue in sanity.sh, <a href="https://jira.whamcloud.com/browse/LU-748" title="sanity test 220 must be placed in SLOW set" class="issue-link" data-issue-key="LU-748"><del>LU-748</del></a>. Test must be moved to SLOW group at least, but ideally all such tests must use smaller OST specially created for testing or special OBD_FAIL_LOC which would simulate OST small size.</p> <p>ost-pools test 14 passed on Lustre v1_8_7_WC1_RC1 in 1850s:<br/> <a href="https://maloo.whamcloud.com/test_sets/a1a53668-f4cf-11e0-908b-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a1a53668-f4cf-11e0-908b-52540025f9af</a></p> <p>The test output showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Filling up OST0 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 200.484 seconds, 14.4 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 179.428 seconds, 16.1 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 160.244 seconds, 18.0 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 166.279 seconds, 17.4 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 159.178 seconds, 18.2 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 160.844 seconds, 18.0 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 177.44 seconds, 16.3 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 187.56 seconds, 15.4 MB/s 2756+0 records in 2756+0 records out 2889875456 bytes (2.9 GB) copied, 170.918 seconds, 16.9 MB/s UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 7498624 443048 6555576 6% /mnt/lustre[MDT:0] lustre-OST0000_UUID 30192956 25817228 2846080 90% /mnt/lustre[OST:0] lustre-OST0001_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:1] lustre-OST0002_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:2] lustre-OST0003_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:3] lustre-OST0004_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:4] lustre-OST0005_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:5] lustre-OST0006_UUID 30192956 454760 28222820 2% /mnt/lustre[OST:6] filesystem summary: 211350692 28545788 172183000 14% /mnt/lustre </pre> </div></div> <p>So, the OST size was about 30G.</p> <p>Chris, does the "OSTSIZE" variable has different values on different vm clusters? Could you please adjust the size so that the test would not timeout and skipped?</p> <p>FYI, due to the timeout of test 14, the remaining ost-pools subtests were not run.</p> <p>Has anybody rerun the missing tests?</p> <p>I will rerun the missing tests for both lu-797 and lu-798 and keep you updated.</p> <p>the remaining tests of ost-pools pass: <a href="https://maloo.whamcloud.com/test_sets/caa784fe-0653-11e1-b578-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/caa784fe-0653-11e1-b578-52540025f9af</a></p> <p>It looks like the failing tests on master are creating 10x 14GB files (consistent with 150GB OST size) and the passing 1.8.7 tests are successfully creating 10x 2.7GB files in 1850s. That means the tests on master would need at least 9500s to pass. That is completely unreasonable for a subtest like this, which should complete in a minute or less (given its low importance compared to other tests).</p> <p>ost-pools.sh test_14 is testing the QOS space balancing for OST pools, which is similar to sanity.sh test_116 is doing for QOS space balance without pools. It seems that test_116 is just being skipped when the OST is too large.</p> <p>Probably the best solution is to have an OBD_FAIL_CHECK() that can change the os_bfree number at the fsfilt or OSD level for the OST index as given in fail_val. Even better might be fail_val = "(free_fraction) << 16 | ost_idx" where free_fraction is the 1/256 of the free space (which gives about 0.5% granularity) so that the fail_loc can be useful in other tests.</p> <p>That will avoid all of the IO in these tests and make them MUCH faster. It would be worthwhile to investigate other tests that need to fill up the filesystem to see if they can similarly be sped up.</p> <p>Another failure:</p> <p><a href="https://maloo.whamcloud.com/test_sets/bdf73f6c-2dc5-11e1-819b-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bdf73f6c-2dc5-11e1-819b-5254004bbbd3</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/ecef3f7a-41db-11e1-9650-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ecef3f7a-41db-11e1-9650-5254004bbbd3</a> (Orion LDiskFS)</p> <p>I submitted change <a href="http://review.whamcloud.com/2028" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2028</a> to speed up the ost-pools script:</p> <p>For test_14, since it is already reformatting the filesystem, it is just going to format with a smaller OST size. I'd rather prefer that we don't reformat the filesystem at all for these tests, and instead implement an OBD_FAIL_CHECK to allow it to run on any configuration, but this is a simple workaround for now that will reduce the time to run ost-pools by 1.5h on large systems.</p> <p>Also, test_18 was taking a long time because it was trying to create and unlink 3 * 3 * 30000 files, which was also taking a very long time for slow VM systems. Instead, use the createmany feature to run for a fixed time instead of for a specific number of files, so that it can properly assess performance without taking too long or to short on different hardware configurations. This should also reduce the test time by 0.5h or so.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/449/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #449</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Landed for 2.2</p> <p>Looks the test_23b is still timeout: <a href="https://maloo.whamcloud.com/test_sets/8bc6156c-4fef-11e1-9ebc-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8bc6156c-4fef-11e1-9ebc-5254004bbbd3</a></p> <blockquote> <p>== ost-pools test 23b: OST pools and OOS ============================================================= 21:02:13 (1328418133)<br/> running as uid/gid/euid/egid 500/500/500/500, groups:<br/> <span class="error">[touch]</span> <span class="error">[/mnt/lustre/d0_runas_test/f5213]</span><br/> client-29vm3: Pool lustre.testpool created<br/> client-29vm3: OST lustre-OST0000_UUID added to pool lustre.testpool<br/> client-29vm3: OST lustre-OST0003_UUID added to pool lustre.testpool<br/> client-29vm3: OST lustre-OST0006_UUID added to pool lustre.testpool<br/> 1: 32768+0 records in<br/> 32768+0 records out<br/> 34359738368 bytes (34 GB) copied, 1560.75 seconds, 22.0 MB/s<br/> 2: 32768+0 records in<br/> 32768+0 records out<br/> 34359738368 bytes (34 GB) copied, 1700.81 seconds, 20.2 MB/s</p></blockquote> <p>This test (test_23b) adds 3 osts into a pool, and checks if the available space of the pool is less than $MAXFREE, however, it writes file over all osts at the end. We missed creating file in pool?</p> <p>Andreas, could you take a look?</p> <p>You are correct, it appears the output files are not being created in the pool. However, looking back at the original test_23, that was true at that time also, so I'm not sure what the original intent of this test was. Looking further back into Git/CVS to determine if the test ran differently when it was introduced, I didn't see any significant changes to test_23 since 1.8.5</p> <p>Note also that I have a second patch which is skipping test_23b() when SLOW=no, because I don't think this is a very useful test (<a href="http://review.whamcloud.com/2091" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2091</a>).</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/472/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #472</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)<br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/test-framework.sh</li> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)<br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/grey.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: speed up ost-pools tests (Revision eea698c944283b755882d8f504d2fcc8ea371bd8)<br/> <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a> tests: skip ost-pools.sh 23b when SLOW=no (Revision f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212)</p> <p> Result = ABORTED<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=eea698c944283b755882d8f504d2fcc8ea371bd8" class="external-link" target="_blank" rel="nofollow noopener">eea698c944283b755882d8f504d2fcc8ea371bd8</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> <li>lustre/tests/test-framework.sh</li> </ul> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212" class="external-link" target="_blank" rel="nofollow noopener">f7b4054cfc1d30fbbfd56acfe4b5a7a334de8212</a><br/> Files : </p> <ul> <li>lustre/tests/ost-pools.sh</li> </ul> <p>Lustre Tag: v1_8_8_WC1_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/197/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/197/</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel version: 2.6.18-308.4.1.el5)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>ost-pools test 14 timed out: <a href="https://maloo.whamcloud.com/test_sets/23fdf678-a258-11e1-a6cd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/23fdf678-a258-11e1-a6cd-52540035b04c</a></p> <p>Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/86/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/86/</a><br/> Distro/Arch: RHEL6.2/x86_64(server), RHEL6.2/i686(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>ost-pools test 23b timed out: <a href="https://maloo.whamcloud.com/test_sets/7d305f1e-aa33-11e1-971d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d305f1e-aa33-11e1-971d-52540035b04c</a></p> <blockquote><p>Note also that I have a second patch which is skipping test_23b() when SLOW=no, because I don't think this is a very useful test (<a href="http://review.whamcloud.com/2091" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2091</a>).</p></blockquote> <p>During the build/release testing cycle, all of the auster tests are run with SLOW=yes, so the ost-pools test 23b timeout issue still exists with large size OSTs:<br/> <a href="http://tinyurl.com/c4gxcbn" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/c4gxcbn</a></p> <p>More instances occurred during Lustre 2.1.3 RC1 testing:<br/> <a href="https://maloo.whamcloud.com/test_sets/b66fac84-e490-11e1-af05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b66fac84-e490-11e1-af05-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/759997b4-e4be-11e1-af05-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/759997b4-e4be-11e1-af05-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/615958d4-e3ec-11e1-b6d3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/615958d4-e3ec-11e1-b6d3-52540035b04c</a></p> <p>More instances occurred during Lustre 2.1.3 RC2 testing:<br/> <a href="https://maloo.whamcloud.com/test_sets/9c6295ce-eb32-11e1-ba73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9c6295ce-eb32-11e1-ba73-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/68c13076-eb3d-11e1-ba73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/68c13076-eb3d-11e1-ba73-52540035b04c</a></p> <p>Look at the recent failure:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">== ost-pools test 23b: OST pools and OOS ============================================================= 21:49:32 (1345438172) running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f13763] CMD: client-30vm3 lctl pool_new lustre.testpool client-30vm3: Pool lustre.testpool created CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool 2>/dev/<span class="code-keyword">null</span> || echo foo CMD: client-30vm3 lctl pool_add lustre.testpool lustre-OST[0000-0006/3] client-30vm3: OST lustre-OST0000_UUID added to pool lustre.testpool client-30vm3: OST lustre-OST0003_UUID added to pool lustre.testpool client-30vm3: OST lustre-OST0006_UUID added to pool lustre.testpool CMD: client-30vm6.lab.whamcloud.com lctl get_param -n lov.lustre-*.pools.testpool | sort -u | tr <span class="code-quote">'\n'</span> <span class="code-quote">' '</span> 1: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1656.82 seconds, 20.7 MB/s 2: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1761.27 seconds, 19.5 MB/s 3: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1921.64 seconds, 17.9 MB/s 4: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 1941.35 seconds, 17.7 MB/s 5: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 2098.63 seconds, 16.4 MB/s 6: 32768+0 records in 32768+0 records out 34359738368 bytes (34 GB) copied, 2030.07 seconds, 16.9 MB/s </pre> </div></div> <p>The MAXFREE should be (2000000(KB) * 7(osts)) / (1024 * 1024) = 13GB, which means the available space should be less than 13GB in this pool, however, tests writes much more than the it and didn't fail with ENOSPC. I think we'd add more debug information in the tests, and also break the writting whenever written bytes is greater than available space.</p> <p><a href="http://review.whamcloud.com/#change,4474" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4474</a></p> <p>remove the space of "lfs df" output interllay: <a href="http://review.whamcloud.com/4705" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4705</a></p> <blockquote><p>remove the space of "lfs df" output interllay: <a href="http://review.whamcloud.com/4705" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4705</a></p></blockquote> <p>b2_1 port is at <a href="http://review.whamcloud.com/#change,4784" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4784</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/148" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/148</a><br/> Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32-279.14.1.el6)<br/> Network: TCP (1GigE)</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/9b37618e-41db-11e2-adcf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9b37618e-41db-11e2-adcf-52540035b04c</a></p> <blockquote><p><a href="http://review.whamcloud.com/#change,4474" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4474</a></p></blockquote> <p>b2_1 port is at <a href="http://review.whamcloud.com/4831" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4831</a></p> <p>Fix landed for 2.1.4 and 2.4</p> <p>The related patches for b1_8 are merged into <a href="http://review.whamcloud.com/4898" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4898</a></p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/236/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/236/</a><br/> Distro/Arch: RHEL5.8/x86_64<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes<br/> OSTSIZE=149723171</p> <p>The ost-pools test 14 timed out: <a href="https://maloo.whamcloud.com/test_sets/f8ded3ce-5187-11e2-bbc3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f8ded3ce-5187-11e2-bbc3-52540035b04c</a></p> Duplicate LU-1001 LU-876 Related Development Rank 1|hzv7en: Rank (Obsolete) 4734 Severity