https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3878 Lustre <p>sanity-benchmark test fsx failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-benchmark test fsx: fsx ==================================================================== 22:15:58 (1378185358) debug=0 Using: fsx -c 50 -p 1000 -S 29278 -P /tmp -l 206139 -N 100000 /mnt/lustre/f0.fsxfile Chance of close/open is 1 in 50 Seed set to 29278 truncating to largest ever: 0xd3af /usr/lib64/lustre/tests/sanity-benchmark.sh: line 186: 12471 Bus error $CMD sanity-benchmark test_fsx: @@@@@@ FAIL: fsx failed </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/becb9218-14ef-11e3-ac48-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/becb9218-14ef-11e3-ac48-52540035b04c</a></p> <p>This is a regression on Lustre b2_4 branch.</p> <br/> Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64 + FC18/x86_64 (Server + Client)<br/> LU-3878

sanity-benchmark test fsx: Bus error

Bug Major Resolved Not a Bug Oleg Drokin Jian Yu Wed, 4 Sep 2013 07:57:34 +0000 Wed, 13 Oct 2021 02:43:13 +0000 Sat, 23 Nov 2013 21:39:05 +0000 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 0 8 <p>This looks related to <a href="https://jira.hpdd.intel.com/browse/LU-2909" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-2909</a> an earlier 2.4 blocker. </p> <p>Sorry, it seems <a href="https://jira.whamcloud.com/browse/LU-2909" title="Failure on test suite sanity-benchmark test_fsx: fsx bus error, core dumped" class="issue-link" data-issue-key="LU-2909"><del>LU-2909</del></a> is closed, so presumably this is a new issue.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>sanity-benchmark test fsx also hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/e004601a-1556-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e004601a-1556-11e3-8938-52540035b04c</a></p> <p>FYI, here is the query result of sanity-benchmark test_fsx with "FAIL" status on Lustre b2_4 branch:<br/> <a href="http://tinyurl.com/kxheu43" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/kxheu43</a></p> <p>By searching on Maloo, I found that test fsx passed on FC18 on Lustre b2_4 build #40 and previous builds. Builds #41, #42, #43 were not tested on FC18. It seems that the culprit is in build #42.</p> <p>After patch <a href="http://review.whamcloud.com/7481" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7481</a> was reverted from Lustre b2_4 branch, the failure did not occur on Lustre 2.4.1 RC2.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>sanity-benchmark test fsx hit the same failure again:<br/> <a href="https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c</a></p> <p>Is this only happening on zfs?</p> <p>Only on b2_4, but not on master?</p> <p>Here is the search result on Maloo:<br/> <a href="http://tinyurl.com/ozo5c7a" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/ozo5c7a</a></p> <p>The failure occurred not only on zfs and b2_4, but also on ldiskfs and master/b2_5:<br/> <a href="https://maloo.whamcloud.com/test_sets/87757614-44d1-11e3-8c03-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/87757614-44d1-11e3-8c03-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/6f3d0b5e-4cc7-11e3-826a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6f3d0b5e-4cc7-11e3-826a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/2582fa8c-3bbf-11e3-b062-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2582fa8c-3bbf-11e3-b062-52540035b04c</a></p> <p>The failure of fax is probably a fallout of the previous failure on iozone. It used up all disk spaces on the OSTs, so there is no any grants on client which made mkwrite() fail.</p> <p>But why does it stop after the patches are reverted?<br/> Also I don't think we see any enospace errors for fsx runs in the logs?</p> <p>the previous iozone run used up all spaces.</p> <p>I can't connect this symptom to that patch. But from what I have seen so far, you reverted that patch on Sep 24 but it still occurred after that.</p> <p>Yes, the failure still occurred on the latest Lustre b2_4 branch with FSTYPE=zfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/17d950e4-58ba-11e3-83d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/17d950e4-58ba-11e3-83d7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/4767b964-57ce-11e3-8d5c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4767b964-57ce-11e3-8d5c-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f21a0152-4ab1-11e3-8252-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f21a0152-4ab1-11e3-8252-52540035b04c</a></p> <p>In the above test reports, all of the iozone tests failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>write: No space left on device </pre> </div></div> <p>or</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write error No space left on device (rc = -1, len = 4194304) </pre> </div></div> <p>So, it seems that the out of space failure of iozone caused the fsx failure.</p> Duplicate LU-2909 Related LU-6729 Development Rank 1|hzvzvz: Rank (Obsolete) 10059 Severity