https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3852 Lustre <p>This issue was created by maloo for Minh Diep &lt;minh.diep@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/3c70376c-0fbb-11e3-bb21-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/3c70376c-0fbb-11e3-bb21-52540035b04c</a>.</p> <p>The sub-test test_251 failed with the following error:</p> <blockquote> <p>request on sanity-hsm is not @@@@@@</p></blockquote> <p>== sanity-hsm test 251: Coordinator request timeout == 00:59:34 (1377676774)<br/> Purging archive<br/> Starting copytool<br/> lhsmtool_posix --hsm-root /tmp/arc --daemon --bandwidth 1 /mnt/lustre<br/> dd: writing `/mnt/lustre2/d0.sanity-hsm/d251/f.sanity-hsm.251': No space left on device<br/> 93+0 records in<br/> 92+0 records out<br/> 96620544 bytes (97 MB) copied, 16.7614 s, 5.8 MB/s<br/> CMD: client-27vm3,client-27vm4,client-27vm5,client-27vm6.lab.whamcloud.com /usr/sbin/lctl dk &gt; /logdir/test_logs/2013-08-27/lustre-reviews-el6-x86_64-<del>review</del>-2_4_1_<em>17733</em>_-70153022088660-162844/sanity-hsm.test_251.debug_log.\$(hostname -s).1377676791.log;<br/> dmesg &gt; /logdir/test_logs/2013-08-27/lustre-reviews-el6-x86_64-<del>review</del>-2_4_1_<em>17733</em>_-70153022088660-162844/sanity-hsm.test_251.dmesg.\$(hostname -s).1377676791.log<br/> CMD: client-27vm3 /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled<br/> mdt.lustre-MDT0000.hsm_control=disabled<br/> Info required for matching: sanity-hsm 251</p> LU-3852

sanity-hsm test_251: client26-vm "dd: no space left on device"

Technical task LU-3647 Blocker Resolved Fixed Bruno Faccini Maloo HSM Wed, 28 Aug 2013 17:04:51 +0000 Sun, 3 Apr 2016 03:33:54 +0000 Wed, 7 Jan 2015 09:46:37 +0000 Lustre 2.5.0 Lustre 2.7.0 0 14 <p>patch is at <a href="http://review.whamcloud.com/7484" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7484</a></p> <p>More instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/faa96ad6-0f85-11e3-9bce-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/faa96ad6-0f85-11e3-9bce-52540035b04c</a></p> <p>Another instance just incase it's needed. <a href="https://maloo.whamcloud.com/test_sets/9a710002-0fcb-11e3-a63c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9a710002-0fcb-11e3-a63c-52540035b04c</a></p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/a4ed5d50-189f-11e3-aa54-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a4ed5d50-189f-11e3-aa54-52540035b04c</a></p> <p>Reopening tickets due to 'No space left on device' failures seen in sanity-hsm again. All tests fail with error 'request on sanity-hsm is not @@@@@@'. <a href="https://maloo.whamcloud.com/test_sets/16a0a15e-2639-11e3-8d26-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/16a0a15e-2639-11e3-8d26-52540035b04c</a> - tests 28, 104, 110b and 251 and <br/> <a href="https://maloo.whamcloud.com/test_sets/96ba3174-265d-11e3-8d26-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/96ba3174-265d-11e3-8d26-52540035b04c</a> - tests 104, 107, 110b and 251.<br/> <a href="https://maloo.whamcloud.com/test_sets/872eeff2-26b6-11e3-94b1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/872eeff2-26b6-11e3-94b1-52540035b04c</a> - test 28, 30b, 31b and 251</p> <p>Hi James,</p> <p>Does your environment include the patch in <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a>?</p> <blockquote> <p> <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a> tests: remove large files created by HSM test</p> <p> Delete files created by HSM test cases with size bigger than 10M.</p></blockquote> <p>Jinshan</p> <p>Yes, I'm using 2.4.93 build # 1687 and that includes the <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a> patch.</p> <p>After checking with James, it turned out there was an issue with setup. I will close it again.</p> <p>I just looked at a Maloo failure which has test_251 failing this way. Maloo stats indicate only a 50% pass rate right now. Looks like this issue is back.</p> <p>Here is the Maloo failure I was looking at: <a href="https://maloo.whamcloud.com/test_sessions/250897b4-5d1c-11e3-956b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/250897b4-5d1c-11e3-956b-52540035b04c</a></p> <p>Again the root cause of the "sanity-hsm test_251: @@@@@@ FAIL: request on sanity-hsm is not @@@@@@" symptom/failure for this ticket, is the "dd: writing `/mnt/lustre2/d0.sanity-hsm/d251/f.sanity-hsm.251': No space left on device" error.</p> <p>Since 2013-09-25 21:03:00, date of last problem's occurrence before Change #7484 has landed, there are only 5 occurrences reported by Maloo stats since 2013-12-05 13:03:17, 2 on client-26vm2 and 3 on client-26vm6.</p> <p>More occurrences and still/only on client-26vm* where Lustre file-system size appear much lower than on other Nodes and likely to fill when creating a big/103MB file in test_251 :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bruno@brent:~$ ssh root@client-26vm6 df /mnt/lustre root@client-26vm6's password: Filesystem 1K-blocks Used Available Use% Mounted on client-26vm3@tcp:/lustre 1464484 191516 1194876 14% /mnt/lustre bruno@brent:~$ ssh root@client-27vm6 df /mnt/lustre root@client-27vm6's password: Filesystem 1K-blocks Used Available Use% Mounted on client-27vm3@tcp:/lustre 14449456 797740 12917724 6% /mnt/lustre bruno@brent:~$ </pre> </div></div> <p>Opened TEI-1289 for this issue.</p> <p>This subtest has been disabled at the autotest level, so a regular test run will skip it. You need to explicitly request testing on the subtest - hopefully Test-Parameters works. </p> <p>Oops, sorry but I think I forgot to indicate that I have created TEI-1289 to address client-26vm* very small sized Lustre filesystem issue.</p> <p>Andreas, I am not sure that test_251 has been "fully" disabled at the autotest level because I still see runs of it for recent patch submissions, and there has been at least one more failure for this same issue reported on December 28th at <a href="https://maloo.whamcloud.com/test_sets/87584406-6fbd-11e3-9a1b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/87584406-6fbd-11e3-9a1b-52540035b04c</a>.</p> <p>Thus, I will raise priority of TEI-1289 to get some update.</p> <p>the sanity-hsm test_251 may only be disabled for tests on master, which is ok I think. </p> <p>Could you please submit a patch to add it to ALWAYS_EXCEPT in the script. When that lands we can remove it from the autotest config so that it will be possible to re-enable it for any patch that is trying to fix the problem. </p> <p>Patch to disable test_251 internally in sanity-hsm has been pushed at <a href="http://review.whamcloud.com/9014" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9014</a>.<br/> When it will land, I suppose I will need to open a TEI to request current auto-tests config settings to be reversed, like for similar situation with <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a>/TEI-570/TEI-1041 ?</p> <p>Requested Gerrit gate-keeper to land patch #9014, next actions will be to re-allow test_251 in autotest config (thru a new TEI?), and then to push a patch that find a way to handle this "small fs size vs file big enough for timing need" requirement, and also to re-allow test_251 (and also test_<span class="error">&#91;200,221,223b&#93;</span> disabled for the same issue in <a href="https://jira.whamcloud.com/browse/LU-4178" title="Test failure on test suite sanity-hsm, subtest test_200" class="issue-link" data-issue-key="LU-4178"><del>LU-4178</del></a>) in same patch.</p> <p>Test has been retriggered: unclear what happened to results</p> <p>Patch #9014 to disable test_251 internally in sanity-hsm has landed. So now, will be able to work on a definitive patch to strengthen/protect test_251 vs too small-sized Lustre FS.</p> <p>Sorry to have been soooo late on this ...<br/> Patch to strengthen sanity-hsm sub-tests, using large files and easy victims of auto-tests configurations with reduced Lustre FS size, is at <a href="http://review.whamcloud.com/12456" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12456</a>.</p> <p>I forgot to indicate that change #12456 also re-enables sanity-hsm/test_<span class="error">&#91;200,221,223b,251&#93;</span> sub-tests.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12456/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12456/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a> tests: skip tests with large file when no room<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3a63fe4aa37ed72dc2f9a2c85c35035c2c2619ba</p> <p>Patch has landed.</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13803" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13803</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3852" title="sanity-hsm test_251: client26-vm &quot;dd: no space left on device&quot;" class="issue-link" data-issue-key="LU-3852"><del>LU-3852</del></a> tests: skip tests with large file when no room<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: d6b1797ac0e0f9ba89688e1fcb986406b65e3418</p> Blocker Related LU-6055 LU-4178 Development Rank 1|hzvzjb: Rank (Obsolete) 9973