https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5765 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>In sanity.sh test_123a and test_123b a large number of errors are being reported when "rm -r" is running:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity2': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity5': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity8': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity11': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity12': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity16': No such file or directory rm: cannot remove `/mnt/lustre/d123a.sanity/f123a.sanity18': No such file or directory </pre> </div></div> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/186592e2-5577-11e4-8542-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/186592e2-5577-11e4-8542-5254006e85c2</a>.</p> <p>Info required for matching: sanity 123a<br/> Info required for matching: sanity 123b</p> LU-5765

sanity test_123a test_123b: rm: no such file or directory

Bug Critical Resolved Fixed Isaac Huang Maloo HB zfs Fri, 17 Oct 2014 20:26:17 +0000 Wed, 14 Jan 2015 18:32:17 +0000 Wed, 14 Jan 2015 18:32:17 +0000 Lustre 2.7.0 0 8 <p>Happened on lustre-rsync-test test_2b on review-dne-part-1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d99c68e6-5653-11e4-b972-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d99c68e6-5653-11e4-b972-5254006e85c2</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e28d061c-5b13-11e4-9c62-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e28d061c-5b13-11e4-9c62-5254006e85c2</a></p> <p>Another failure <a href="https://testing.hpdd.intel.com/test_sets/729ab070-5b5e-11e4-95e9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/729ab070-5b5e-11e4-95e9-5254006e85c2</a></p> <p>Di, Fan Yong,<br/> it looks like readdir is failing for some reason (e.g. looping and returning the same entry hundreds of times?) and that is why "rm" is returning an error for all of the later entries. Is this statahead, or DNE going wrong? Unfortunately, we don't have any debug logs to tell us what is going on.</p> <p>It looks like these failures are all happening on ZFS. I don't think the lustre-rsync-test test_2b looks like the same symptom AFAICS.</p> <p>Isn't there another bug open with ZFS directory iteration broken? Maybe they are related?</p> <p>I repeated "sanity --only 123a,123b" for 100 times, and couldn't reproduce it. I was using build lustre-b2_5/96/ with 1 OSS (2 OSTs) 1 MDS and 1 client, any other build/configuration I should try?</p> <p>Another 50 repetitions, still can't reproduce, although the eagle VMs I used all had just 1 CPU.</p> <p>Isaac, I think you should be testing with master and ZFS 0.6.3, and not b2_5, since all of the failures I've seen have been on master so far.</p> <p>Maybe a patch should be landed to sanity.sh test_123<span class="error">&#91;ab&#93;</span> to help diagnose the problem if it happens again under review testing? For example, doing an "ls" of the directory after the test is done, always running "rm" under strace and logging it to a file that is attached to Maloo so that we can see if the problem is in readdir() data returned to userspace, etc. There have been 8 failures in the last 165 review-zfs tests in the past week, so I think if a patch is landed to sanity.sh on master it should be possible to get more information within a day or two. I don't think just running review testing on the patch itself is likely to see problems, unless we run sanity.sh 20x in a loop with <tt>Test-Parameters:</tt>.</p> <p>Isaac, another possible condition is that our Maloo test clusters run a lots VMs on single node, so the system load on Maloo clusters should be much higher than our personal test environment. So if we can simulate the similar test environment as Maloo clusters do, then it may be helpful to reproduce the issue locally.</p> <p>With a build on master, I'm now able to reproduce it almost 100%. I'll look into it.</p> <p>In one test today, test_123a took 18127 seconds to complete (more than 100x the usual time on the same hw), and there were 11575754 "no such file" errors in the test log.</p> <p>Is it possible that this problem is the same as <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> and was fixed by <a href="http://review.whamcloud.com/12904" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12904</a> ?</p> <p>The most recent failure <a href="https://testing.hpdd.intel.com/test_sets/97ea0fd0-84f6-11e4-a60f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/97ea0fd0-84f6-11e4-a60f-5254006e85c2</a> was on a patch based on a tree that doesn't contain the <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a> fix.</p> <p>Before that, the most recent test failure was <a href="https://testing.hpdd.intel.com/test_sets/863b9434-fcb2-11e2-9222-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/863b9434-fcb2-11e2-9222-52540035b04c</a> on 2014-08-03.</p> <p>Saw this again on a recent patch run:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e55f7412-881b-11e4-aa28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e55f7412-881b-11e4-aa28-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/c69c83ca-87f9-11e4-a70f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c69c83ca-87f9-11e4-a70f-5254006e85c2</a></p> <p>This might also be related to <a href="https://jira.whamcloud.com/browse/LU-6101" title="sanity test_24A: Can not delete directories" class="issue-link" data-issue-key="LU-6101"><del>LU-6101</del></a>, which also has a patch. </p> <p>Haven't seen this again for the past 4 weeks. </p> <p>Closing it again. Maybe <a href="https://jira.whamcloud.com/browse/LU-6101" title="sanity test_24A: Can not delete directories" class="issue-link" data-issue-key="LU-6101"><del>LU-6101</del></a> will fix the final trigger. </p> Duplicate LU-3573 Related LU-6101 Development Rank 1|hzwyx3: Rank (Obsolete) 16184 Severity