https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-798 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/6924c914-00ed-11e1-bb4f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6924c914-00ed-11e1-bb4f-52540025f9af</a>.<br/> I am not sure if this is duplicate with <a href="https://jira.whamcloud.com/browse/LU-758" title="Test failure on test suite performance-sanity, subtest test_8" class="issue-link" data-issue-key="LU-758"><del>LU-758</del></a></p> LU-798

Test failure on test suite performance-sanity,subtest test_5

Bug Blocker Resolved Cannot Reproduce Zhenyu Xu Maloo Thu, 27 Oct 2011 19:47:59 +0000 Wed, 13 Jul 2016 20:57:27 +0000 Wed, 13 Jul 2016 20:57:27 +0000 Lustre 2.1.2 Lustre 2.1.3 Lustre 2.1.4 Lustre 1.8.8 0 8 <p>Bobijam will look into this one</p> <p>from the dmesg log, I don't see any error happens during the test except test_5 took almost an hour, I suspect its 3600 seconds test limit interrupt the test.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>client dmesg</b></div><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == performance-sanity test 5: lookup rate 10M file dir =============================================== 09:45:59 (1319733959) Lustre: DEBUG MARKER: ===== mdsrate-lookup-1dir.sh ====== Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: ===== mdsrate-lookup-1dir.sh Test preparation: creating 1000000 files. Lustre: DEBUG MARKER: ===== mdsrate-lookup-1dir.sh Lustre: DEBUG MARKER: ===== mdsrate-lookup-1dir.sh Lustre: DEBUG MARKER: == mdsrate-lookup-1dir mdsrate-lookup-1dir.sh: test complete, cleaning up ============================ 10:33:31 (1319736811) SysRq : Show State sibling task PC pid father child younger older init S ffff81007e3567e0 0 1 0 2 (NOTLB) ffff81007ff95a28 0000000000000082 00000000000280d2 ffff810000017c10 ffff81007ff857a0 000000000000000a ffff81007ff857a0 ffff81007e3567e0 00000aabaa90b8db 0000000000006e20 ffff81007ff85988 0000000088031cb7 </pre> </div></div> <p>I checked maloo for some passed performance-sanity test_5 to get a rough understanding how much time was needed for successfull test.</p> <p>This test case <a href="https://maloo.whamcloud.com/sub_tests/9e94a664-f710-11e0-a451-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/9e94a664-f710-11e0-a451-52540025f9af</a> takes 2638s, while it only test with 433075 files (by checking test script, I think it was set by NUM_FILES environment variable)</p> <p>This test case <a href="https://maloo.whamcloud.com/sub_tests/116bdd9a-e467-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/116bdd9a-e467-11e0-9909-52540025f9af</a> takes 2493s with 452393 files.</p> <p>1M files test cases all timed out on cleaning up phase. </p> <p>performance-sanity test 5 passed on v1_8_7_WC1_RC1 in 2692s with 438805 files:<br/> <a href="https://maloo.whamcloud.com/test_sets/eb31d5ac-f4cf-11e0-908b-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eb31d5ac-f4cf-11e0-908b-52540025f9af</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 5000040 52 4999988 0% /mnt/lustre[MDT:0] lustre-OST0000_UUID 445440 2202 443238 0% /mnt/lustre[OST:0] lustre-OST0001_UUID 445440 88 445352 0% /mnt/lustre[OST:1] lustre-OST0002_UUID 445440 89 445351 0% /mnt/lustre[OST:2] lustre-OST0003_UUID 445440 89 445351 0% /mnt/lustre[OST:3] lustre-OST0004_UUID 445440 89 445351 0% /mnt/lustre[OST:4] lustre-OST0005_UUID 445440 89 445351 0% /mnt/lustre[OST:5] lustre-OST0006_UUID 445440 89 445351 0% /mnt/lustre[OST:6] filesystem summary: 5000040 52 4999988 0% /mnt/lustre ===== mdsrate-lookup-1dir.sh Test preparation: creating 438805 files. </pre> </div></div> <p>The number of files was calculated as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdsrate-lookup-1dir.sh: NUM_FILES=${NUM_FILES:-1000000} IFree=$(mdsrate_inodes_available) if [ $IFree -lt $NUM_FILES ]; then NUM_FILES=$IFree fi test-framework.sh: inodes_available () { local IFree=$($LFS df -i $MOUNT | grep ^$FSNAME | awk '{print $4}' | sort -un | head -1) || return 1 echo $IFree } mdsrate_inodes_available () { local min_inodes=$(inodes_available) echo $((min_inodes * 99 / 100)) } </pre> </div></div> <p>So, it depends on the minimum number of free inodes on MDT and OST. For RC1, the OSTSIZE was about 30G and the min IFree was about 443238, however, for RC2, the OSTSIZE on the vm cluster was about 150G and the min IFree was about 2246430:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 5000040 52 4999988 0% /mnt/lustre[MDT:0] lustre-OST0000_UUID 2257920 11490 2246430 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 2257920 89 2257831 0% /mnt/lustre[OST:1] lustre-OST0002_UUID 2257920 89 2257831 0% /mnt/lustre[OST:2] lustre-OST0003_UUID 2257920 88 2257832 0% /mnt/lustre[OST:3] lustre-OST0004_UUID 2257920 89 2257831 0% /mnt/lustre[OST:4] lustre-OST0005_UUID 2257920 89 2257831 0% /mnt/lustre[OST:5] lustre-OST0006_UUID 2257920 89 2257831 0% /mnt/lustre[OST:6] filesystem summary: 5000040 52 4999988 0% /mnt/lustre ===== mdsrate-lookup-1dir.sh Test preparation: creating 1000000 files. </pre> </div></div> <p>So, this is the similar issue as <a href="https://jira.whamcloud.com/browse/LU-797" title="Test failure on test suite ost-pools, subtest test_14, test_18, test_23" class="issue-link" data-issue-key="LU-797"><del>LU-797</del></a>, which is caused by large OSTSIZE and limited timeout value. </p> <p>Chris, could you please take a look at this issue and make some adjustment on the OSTSIZE or timeout value? Thanks!</p> <p>&gt; Chris, could you please take a look at this issue and make some adjustment on the OSTSIZE or timeout value? Thanks!</p> <p>Bobijam, it would be better to make such requests via a TT JIRA ticket.</p> <p>the remaining tests of performance-sanity are pass: <a href="https://maloo.whamcloud.com/test_sets/7d4cb962-0668-11e1-9433-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d4cb962-0668-11e1-9433-52540025f9af</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/194/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/194/</a><br/> Distro/Arch: RHEL5.8/x86_64 (kernel 2.6.18-308.4.1.el5)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/b8d82eb2-9b50-11e1-a0a0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b8d82eb2-9b50-11e1-a0a0-52540035b04c</a></p> <p>Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/86/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/86/</a><br/> Distro/Arch: RHEL6.2/x86_64(server), RHEL6.2/i686(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>performance-sanity test 6 timed out: <a href="https://maloo.whamcloud.com/test_sets/9b7b720a-aa34-11e1-971d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9b7b720a-aa34-11e1-971d-52540035b04c</a></p> <p>More instances occurred while testing Lustre 2.1.3 RC2:<br/> <a href="https://maloo.whamcloud.com/test_sets/b5722254-eb33-11e1-ba73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b5722254-eb33-11e1-ba73-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/9c1e8436-eb3e-11e1-ba73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9c1e8436-eb3e-11e1-ba73-52540035b04c</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/148" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/148</a><br/> Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32-279.14.1.el6)<br/> Network: TCP (1GigE)</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/eba241ba-41dc-11e2-adcf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eba241ba-41dc-11e2-adcf-52540035b04c</a></p> <p>b2_1 patch tracking at <a href="http://review.whamcloud.com/4830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4830</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre> LU-798 test: reduce performance test file number The performance test takes too much time deleting all the test files which timedout the test suite. Test-Parameters: envdefinitions=SLOW=yes,ENABLE_QUOTA=yes testlist=performance-sanity </pre> </div></div> <p>Lustre Tag: v2_1_4_RC1</p> <p><a href="https://maloo.whamcloud.com/test_sets/07afddb8-4831-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/07afddb8-4831-11e2-8cdc-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/af0b031e-482a-11e2-a406-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/af0b031e-482a-11e2-a406-52540035b04c</a></p> <p>From the historical Maloo reports, I found that all of the passed performance-sanity test runs were performed with about 450000 files, while all of the timeouted performance-sanity test runs were performed with 1000000 files due to large OSTSIZE.</p> <p>Hello Chris,</p> <p>Oleg suggested to adjust the length of time that is allocated to the performance-sanity test so that it has time to complete. Could you please increase the autotest timeout value for the performance-sanity test? Thanks.</p> <p>performance-sanity is set to 2 hours for test_4, test_5 and test_7 so I fear the code might be broken in autotest. I need to figure out how to investigate.</p> <p>OK.</p> <p>What confused me here is that this bug dates back to 2011 and so many of the examples have been fixed a long time ago.</p> <p>I've increased the timeout for test_7 to 7200 which makes it like 5,6 and 8.</p> <p>Looking at the recent examples this issue should now be fixed.</p> <p>Hi Chris,</p> <p>The performance-sanity test timed out on the latest b1_8 build #241. This time it's sub-test 8:</p> <p><a href="https://maloo.whamcloud.com/test_sets/79fc558c-5ce9-11e2-afaf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/79fc558c-5ce9-11e2-afaf-52540035b04c</a></p> <p>I did not find any Lustre error messages in the logs. So, it's still due to low performance and needs more time.</p> <p>Closing this old bug.</p> Related Development Rank 1|hzvdi7: Rank (Obsolete) 5794 Severity