https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1233 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/7722bc68-70e8-11e1-a89e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7722bc68-70e8-11e1-a89e-5254004bbbd3</a>.</p> <p>The sub-test test_compilebench failed with the following error:</p> <blockquote> <p>compilebench failed: 1</p></blockquote> <p>Info required for matching: parallel-scale compilebench</p> LU-1233

Test failure on test suite parallel-scale, subtest test_compilebench,no space left

Bug Blocker Resolved Fixed Jian Yu Maloo Mon, 19 Mar 2012 14:22:44 +0000 Tue, 31 Dec 2013 15:55:14 +0000 Wed, 11 Dec 2013 14:14:39 +0000 Lustre 2.3.0 Lustre 2.1.5 Lustre 1.8.9 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.1.6 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 7 <p>another no space left: <a href="https://maloo.whamcloud.com/test_sets/8653c500-76ca-11e1-ae2e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8653c500-76ca-11e1-ae2e-5254004bbbd3</a></p> <p>More instances:<br/> <a href="https://maloo.whamcloud.com/test_sets/e19f266e-1474-11e2-8ca0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e19f266e-1474-11e2-8ca0-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/0631ab8c-1475-11e2-8ca0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0631ab8c-1475-11e2-8ca0-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/99581f3c-1474-11e2-8ca0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/99581f3c-1474-11e2-8ca0-52540035b04c</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/139" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/139</a><br/> Network: o2ib</p> <p>performance-sanity: <a href="https://maloo.whamcloud.com/test_sets/c2a8267c-3ba0-11e2-b98e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c2a8267c-3ba0-11e2-b98e-52540035b04c</a><br/> parallel-scale: <a href="https://maloo.whamcloud.com/test_sets/141e9cfc-3ba1-11e2-b98e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/141e9cfc-3ba1-11e2-b98e-52540035b04c</a></p> <p>Minh</p> <p>Could you please look at this issue? We need identify the OSTSIZE used for the IB test cluster and do a comparison with the one set on the TCP test cluster so that the settings in autotest for IB clusters can be adjusted accordingly.</p> <p>Thanks</p> <p>Peter</p> <p>Chris confirmed that the OST devices on the ib clusters are only 2G; which means 2G * 7 OST = 14G. This is very small filesystem. Compare to 142G * 7 OST in TCP.</p> <p>I see that the pv on the cluster is</p> <p><span class="error">&#91;root@client-21-ib ~&#93;</span># pvdisplay<br/> &#8212; Physical volume &#8212;<br/> PV Name /dev/sda4<br/> VG Name lvm-OSS<br/> PV Size 207.93 GiB / not usable 4.84 MiB<br/> Allocatable yes <br/> PE Size 4.00 MiB<br/> Total PE 53230<br/> Free PE 49516<br/> Allocated PE 3714<br/> PV UUID ytFmgj-Xy6w-Xa4Z-fYHo-cvvI-UzJg-lvPMYX</p> <p>I suggest we increase the lv for oct to use bigger space. How about 10G each?</p> <p>Thanks Minh. Chris can you please comment?</p> <p>Lustre Server: v2_1_4_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/159/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/159/</a></p> <p>Lustre Client: 1.8.8-wc1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/198" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/198</a></p> <p>Distro/Arch: RHEL5.8/x86_64<br/> Network: IB (in-kernel OFED)</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/431b1a6a-479c-11e2-876e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/431b1a6a-479c-11e2-876e-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f2986dea-479b-11e2-876e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f2986dea-479b-11e2-876e-52540035b04c</a></p> <p>In performance-sanity, after creating large number files hit out of space issue, those files were not unlinked/removed successfully. So, the test script also needs to be improved.</p> <p>Lustre Client: v2_1_4_RC1<br/> Lustre Server: 2.1.3<br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: IB (in-kernel OFED)</p> <p><a href="https://maloo.whamcloud.com/test_sets/d03b0306-487d-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d03b0306-487d-11e2-8cdc-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/362d462e-487e-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/362d462e-487e-11e2-8cdc-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/709191e4-487e-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/709191e4-487e-11e2-8cdc-52540035b04c</a></p> <p>This issue is blocking the Lustre 2.1.4 release testing on IB network in autotest runs.</p> <p>The OST size under autotest is the same for IB or TCP why would this issue only effect IB if it is a OST size issue?</p> <p>Are TCP runs using VM or real HW?</p> <p>Lustre Branch: b1_8<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/249" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/249</a><br/> Distro/Arch: RHEL5.8/x86_64(server), RHEL6.3/x86_64(client)<br/> Network: TCP</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=2097152 OSTSIZE=7416428 </pre> </div></div> <p>The compilebench tests in parallel-scale</p> {,-nfsv3,-nfsv4} <p> all failed with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 23875:0:(filter.c:3459:filter_precreate()) create failed rc = -28 </pre> </div></div> <p>Maloo reports:<br/> parallel-scale: <a href="https://maloo.whamcloud.com/test_sets/e021f142-6337-11e2-ae8b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e021f142-6337-11e2-ae8b-52540035b04c</a><br/> parallel-scale-nfsv3: <a href="https://maloo.whamcloud.com/test_sets/85799c1c-6338-11e2-ae8b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/85799c1c-6338-11e2-ae8b-52540035b04c</a><br/> parallel-scale-nfsv4: <a href="https://maloo.whamcloud.com/test_sets/ffb1cbd0-6338-11e2-ae8b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ffb1cbd0-6338-11e2-ae8b-52540035b04c</a></p> <p>With the following values, the same tests passed on the same Lustre b1_8 build over TCP network on RHEL5.8/x86_64 disto/arch (both server and client):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=2097152 OSTSIZE=11311139 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sessions/820df576-6353-11e2-ae8b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/820df576-6353-11e2-ae8b-52540035b04c</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/164" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/164</a><br/> Network: o2ib (in-kernel OFED)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=2097152 OSTSIZE=31061817 </pre> </div></div> <p>performance-sanity: <a href="https://maloo.whamcloud.com/test_sets/ac776a56-68dd-11e2-ac0a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ac776a56-68dd-11e2-ac0a-52540035b04c</a><br/> parallel-scale: <a href="https://maloo.whamcloud.com/test_sets/d2cc2afc-68dd-11e2-ac0a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d2cc2afc-68dd-11e2-ac0a-52540035b04c</a></p> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/186" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/186</a><br/> Distro/Arch: RHEL6.3/x86_64<br/> Network: IB (in-kernel OFED)</p> <p>The issue is still blocking the tests after performance-sanity in full test group from running under IB network configuration:<br/> <a href="https://maloo.whamcloud.com/test_sessions/6f41a40a-8b40-11e2-aa18-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/6f41a40a-8b40-11e2-aa18-52540035b04c</a> </p> <p>Lustre Client: 1.8.9-wc1<br/> Lustre Client Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/258/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/258/</a></p> <p>Lustre Server: v2_1_5_RC1<br/> Lustre Server Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/191/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/191/</a></p> <p>Network: TCP (1GigE)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=2097152 OSTSIZE=149718677 </pre> </div></div> <p>The tests after performance-sanity were affected by the out of space issue:<br/> <a href="https://maloo.whamcloud.com/test_sessions/3bb63464-92c7-11e2-b06e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/3bb63464-92c7-11e2-b06e-52540035b04c</a></p> <p>Dmesg on MDS node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: ===== mdsrate-stat-large.sh Test preparation: creating 1000000 files. LustreError: 21464:0:(mdd_dir.c:1889:mdd_create()) error on stripe info copy -28 LustreError: 21464:0:(mdd_dir.c:1889:mdd_create()) error on stripe info copy -28 Lustre: DEBUG MARKER: /usr/sbin/lctl mark performance-sanity test_8: @@@@@@ FAIL: test_8 failed with 1 </pre> </div></div> <p>Lustre Branch: b2_1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/204" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/204</a><br/> Distro/Arch: RHEL5.9/x86_64<br/> Network: IB (in-kernel OFED)</p> <p>The tests after performance-sanity were affected by the out of space issue:<br/> <a href="https://maloo.whamcloud.com/test_sessions/bebd9a1c-c5c8-11e2-9bf1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/bebd9a1c-c5c8-11e2-9bf1-52540035b04c</a></p> <blockquote><p>In performance-sanity, after creating large number files hit out of space issue, those files were not unlinked/removed successfully. So, the test script also needs to be improved.</p></blockquote> <p>Here is the patch to unlink the files created in performance-sanity.sh through mdsrate-{create,lookup,stat}-*.sh after create/lookup/stat operation fails:<br/> <a href="http://review.whamcloud.com/6483" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6483</a></p> <p>With the above patch, the issue was narrowed that only performance-sanity test 4 hit out of space issue, and the tests after test 4 were not affected.</p> <p>The next step is to figure out why test 4 hits out of space issue over IB network.</p> <p>Lustre Client: 1.8.9-wc1<br/> Lustre Client Build: <a href="http://build.whamcloud.com/job/lustre-b1_8/258/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/258/</a></p> <p>Lustre Server: v2_1_6_RC1<br/> Lustre Server Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/208/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/208/</a></p> <p>Network: TCP (1GigE)</p> <p>performance-sanity test_8 failed with out of space issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/58eec84a-cb8c-11e2-a1fe-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/58eec84a-cb8c-11e2-a1fe-52540035b04c</a></p> <blockquote><p>Here is the patch to unlink the files created in performance-sanity.sh through mdsrate-{create,lookup,stat}-*.sh after create/lookup/stat operation fails: <a href="http://review.whamcloud.com/6483" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6483</a></p></blockquote> <p>The above patch has landed on Lustre b2_1 branch.<br/> Here is the patch for master branch: <a href="http://review.whamcloud.com/8265" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8265</a>. It also needs to be cherry-picked to Lustre b2_5 branch.<br/> And here is the patch for Lustre b2_4 branch: <a href="http://review.whamcloud.com/8289" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8289</a>.</p> <p>Patch landed on Lustre b2_4 branch for 2.4.2 and on master branch for 2.6.0.</p> <p>Can this ticket be closed?</p> <blockquote><p>Can this ticket be closed?</p></blockquote> <p>I'll back-port the patch to Lustre b2_5 branch.</p> <p>Patches have landed to Master. Yu Jian will backport to b2_5 branch.</p> Development Rank 1|hzva6f: Rank (Obsolete) 5184 Severity