https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9872 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/9b7c7e8e-7b5a-4f4d-af09-400c586a8340" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/9b7c7e8e-7b5a-4f4d-af09-400c586a8340</a></p> <p>This issue looks like <a href="https://jira.whamcloud.com/browse/LU-3801" title="Interop 2.1.5&lt;-&gt;2.5 failure on test suite parallel-scale-nfsv3 test_connectathon: can&#39;t sync bigfile" class="issue-link" data-issue-key="LU-3801"><del>LU-3801</del></a>, but it failed with an I/O error rather than no space.</p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>write/read 30 MB file Warning: can't complete test: can't sync bigfile21643: No space left on device special tests failed parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4980:error() = /usr/lib64/lustre/tests/functions.sh:548:run_connectathon() = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:108:test_connectathon() = /usr/lib64/lustre/tests/test-framework.sh:5256:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5295:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5142:run_test() = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:110:main() </pre> </div></div> Trevis2, full<br/> server: RHEL 7.3, ldiskfs, branch master, v2.10.51, b3620<br/> client: RHEL 7.3, branch master, v2.10.51, b3620<br/> LU-9872

parallel-scale-nfsv3 test_connectathon: connectathon failed: 1

Bug Major Resolved Fixed Bob Glossman James Casper Fri, 11 Aug 2017 20:40:41 +0000 Sat, 3 Nov 2018 01:54:18 +0000 Wed, 29 Nov 2017 13:17:45 +0000 Lustre 2.11.0 Lustre 2.11.0 Lustre 2.10.3 0 5 <p>Bob</p> <p>Could you please look into this one? New in the latest master tag and fails 80% of the time.</p> <p>Peter</p> <p>James said</p> <blockquote> <p>This issue looks like <a href="https://jira.whamcloud.com/browse/LU-3801" title="Interop 2.1.5&lt;-&gt;2.5 failure on test suite parallel-scale-nfsv3 test_connectathon: can&#39;t sync bigfile" class="issue-link" data-issue-key="LU-3801"><del>LU-3801</del></a>, but it failed with an I/O error rather than no space.</p></blockquote> <p>But from test log it looks like it is in fact failing with no space:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Warning: can't complete test: can't sync bigfile21643: No space left on device special tests failed </pre> </div></div> <p>Can't say why it is running out of space.</p> <p>The most recent change I can find that seems like it might have an impact is</p> <p><a href="https://jira.whamcloud.com/browse/LU-6900" title="improve parallel-scale-nfs suite" class="issue-link" data-issue-key="LU-6900"><del>LU-6900</del></a> tests: parallel-scale-nfs improvement</p> <p>That landed 6/7/17</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29786" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29786</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9872" title="parallel-scale-nfsv3 test_connectathon: connectathon failed: 1" class="issue-link" data-issue-key="LU-9872"><del>LU-9872</del></a> tests: check space requirements for NFS test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 67172edc5eb09bff9c3cc28e37188375d76af77f</p> <p>parallel-scale-nfsv3 test connectathon started failing with the sync file error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>write/read 30 MB file Warning: can't complete test: can't sync bigfile7456: No space left on device special tests failed parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 </pre> </div></div> <p>starting on August 1, 2017 with master tag 2.10.51 build #3620. Logs for the first failure are at<br/> <a href="https://testing.hpdd.intel.com/test_sets/02ecbd6e-79a0-11e7-8e1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/02ecbd6e-79a0-11e7-8e1f-5254006e85c2</a><br/> Logs for more recent failures are at<br/> <a href="https://testing.hpdd.intel.com/sub_tests/990dadc2-bb71-11e7-9abd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/990dadc2-bb71-11e7-9abd-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/744189ca-ba98-11e7-8afb-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/744189ca-ba98-11e7-8afb-52540065bddc</a></p> <p>parallel-scale-nfsv4 started failing with this error on October 11, 2017 with logs at <a href="https://testing.hpdd.intel.com/sub_tests/e41720d8-af86-11e7-a26c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/e41720d8-af86-11e7-a26c-5254006e85c2</a> </p> <p>I’ve been running this test to figure out how much space test_connectathon needs to run and add code to the script to skip the test when the file system doesn’t have enough space. From what I can tell, the “bigfile” test writes a 30 MB file and the “bigfile2” test writes at 2 and 4 GB boundaries, but I haven’t seen a 2 or 4 GB file created. So, it looks like the largest file written is 30 MB. Yet, this test still fails with ‘No space left on device’ when there is 177152 KB available.</p> <p>I was trying to find the maximum file system size that connectathon would fail with the ‘No space left on device’ error when running the “special” test type. Looking at the connectathon code, it looks like the “bigfile” test should take the most space on the file system by creating a 30M file. So, I started running parallel-scale-nfsv3 on smaller and smaller file systems until I got the ‘no space’ error. </p> <p>At ~96 MB (100925440 bytes) file system, I was able to get on run of parallel-scale-nfsv3 to fail with the no space error, but seven other runs with the same file system ran to completion. So I printed out the amount of free space on the file system before and after each of the connectathon tests, basic (-b), general (-g), special (-s), and lock (–l).</p> <p>For a test run that succeeded, here’s the memory before and after 10 iterations of each test type:<br/> Before –b tests Free space: 100925440 bytes<br/> After –b tests Free space: 96731136 bytes<br/> Before –g tests Free space: 96731136 bytes<br/> After –g tests Free space: 97255424 bytes<br/> Before –s tests Free space: 97255424 bytes<br/> After –s tests Free space: 99090432 bytes<br/> Before –l tests Free space: 99090432 bytes<br/> After –l tests Free space: 100401152 bytes</p> <p>Another run that succeeded:<br/> Before –b tests Free space: 100925440 bytes<br/> After –b tests Free space: 94371840 bytes<br/> Before –g tests Free space: 94371840 bytes<br/> After –g tests Free space: 95158272 bytes<br/> Before –s tests Free space: 95158272 bytes<br/> After –s tests Free space: 49807360 bytes<br/> Before –l tests Free space: 49807360 bytes<br/> After –l tests Free space: 100401152 bytes</p> <p>For a test run that fails in the special test type with the no space error, we see<br/> Before –b tests Free space: 100925440 bytes<br/> After –b tests Free space: 94371840 bytes<br/> Before –g tests Free space: 94371840 bytes<br/> After –g tests Free space: 97255424 bytes<br/> Before –s tests Free space: 97255424 bytes<br/> <span class="error">&#91;Experienced &#39;no space&#39; failure in –s tests&#93;</span><br/> After –s tests Free space: 81264640 bytes</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29786/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29786/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9872" title="parallel-scale-nfsv3 test_connectathon: connectathon failed: 1" class="issue-link" data-issue-key="LU-9872"><del>LU-9872</del></a> tests: modify check space requirements for NFS test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0b8e9558e88930814857c97dfe2394f8c8e24a9a</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30353" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30353</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9872" title="parallel-scale-nfsv3 test_connectathon: connectathon failed: 1" class="issue-link" data-issue-key="LU-9872"><del>LU-9872</del></a> tests: modify check space requirements for NFS test<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: db79de77aec28a0e84f65dcc36b7b60895047887</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30353/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30353/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9872" title="parallel-scale-nfsv3 test_connectathon: connectathon failed: 1" class="issue-link" data-issue-key="LU-9872"><del>LU-9872</del></a> tests: modify check space requirements for NFS test<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 03b330d95434337f3270960f59c62e07a56c5a43</p> Related LU-10689 LU-4905 Development Rank 1|hzzifj: Rank (Obsolete) 9223372036854775807 Severity