Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-479] Test failure on test suite sanity, subtest test_124a https://jira.whamcloud.com/browse/LU-479 Lustre <p>This issue was created by maloo for yujian <yujian@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/282aae9a-a451-11e0-b786-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/282aae9a-a451-11e0-b786-52540025f9af</a>.</p> <p>The sub-test test_124a failed with the following error:</p> <blockquote> <p>No locks dropped in 50s. LRU size: 2003</p></blockquote> LU-479

Test failure on test suite sanity, subtest test_124a

Bug Minor Resolved Cannot Reproduce Lai Siyao Maloo Sun, 3 Jul 2011 23:44:06 +0000 Sun, 28 May 2017 06:58:02 +0000 Sun, 28 May 2017 06:58:01 +0000 Lustre 2.1.0 0 7 <p>More failure instances:<br/> <a href="https://maloo.whamcloud.com/test_sets/4b97a594-a448-11e0-b786-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4b97a594-a448-11e0-b786-52540025f9af</a><br/> <a href="https://maloo.whamcloud.com/test_sets/c597e884-a44f-11e0-b786-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c597e884-a44f-11e0-b786-52540025f9af</a></p> <p>Maloo test logs shows:<br/> LRU=2003<br/> LVF=0</p> <p>Herein LVF should be a big number (around 10000), but it's 0 here, I will print more tunables in the script to find why.</p> <p>This problem is causing quite a lot of sanity test failures lately. Has any investigation been done to determine when this test started failing, and what changes were made around that time?</p> <p>This doesn't always fail, and a possible caused may be autotest environment change (moving from physical machines to VM, and using lvm as backend instead of /dev/sdx).</p> <p>Landed for Lustre 2.1</p> <p>Oops. I think that the patch landed is actually a diagnostic patch rather than a fix so reopening</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/199/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #199</a><br/> <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a> sanity 124a failed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=c4cb5e5f0e316f986eae3b11be4f5d81756b85b6" class="external-link" target="_blank" rel="nofollow noopener">c4cb5e5f0e316f986eae3b11be4f5d81756b85b6</a><br/> Files : </p> <ul> <li>lustre/tests/sanity.sh</li> </ul> <p>Does it make sense to try and reproduce this manually in a vm environment as with LU482?</p> <p>Okay, I will test it manually.</p> <p>Hi Chris,</p> <p>I ran sanity and test_124a for 2 days on client16-vm<span class="error">[1-3]</span>, but couldn't reproduce this failure.</p> <p>Last weekend Toro nodes were reserved, and today these VMs can't be accessed yet, Chris, could you help me?</p> <p>thanks,</p> <ul class="alternate" type="square"> <li>Lai</li> </ul> <p>I've rebuilt the VM's after Sam's work. They should be usable now.</p> <p>Chris</p> <p>In maloo test result, one log message are repeated from the beginning of test_124a:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(ldlm_request.c:1218:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 102336) </pre> </div></div> <p>Because zero SLV is invalid, client pool limit and slv won't be updated.</p> <p>It's strange that MDS replies zero SLV to client, because the only chance obd->obd_pool_slv equals 0 is that ldlm_pool_recalc() is never called. </p> <p>Hi Peter, I couldn't reproduce this failure (for 5 days), and to make debug easier, I'd suggest to enable this test on regression to help collect debug logs.</p> <p>Chris</p> <p>Can you take care of this?</p> <p>Peter</p> <p>Dropping as blocker for now. May re-add if it starts reoccurring regularly.</p> <p>got another hit on <a href="https://maloo.whamcloud.com/test_sets/5ecde210-d96b-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5ecde210-d96b-11e0-8d02-52540025f9af</a></p> <p>It looks like these hit it as well:<br/> <a href="https://maloo.whamcloud.com/test_sessions/7c897254-fdde-11e1-a1b4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/7c897254-fdde-11e1-a1b4-52540035b04c</a> <a href="https://maloo.whamcloud.com/test_sessions/cb9c8b6c-f6e3-11e1-b320-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/cb9c8b6c-f6e3-11e1-b320-52540035b04c</a></p> <p>Close old issue.</p> Development Rank 1|hzv8tz: Rank (Obsolete) 4965 Severity