https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10663 Lustre <p>When run obdfilter-survey it would only do one test at a time. After some debugging I traced the issue to destroy_objects function. Here is the errors.</p> <p>remote_shell localhost lctl --device 17 destroy 444 1<br/> error: destroy: invalid objid '444'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;<br/> remote_shell localhost lctl --device 18 destroy 444 1<br/> error: destroy: invalid objid '444'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;<br/> remote_shell localhost lctl --device 19 destroy 444 1<br/> error: destroy: invalid objid '444'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;<br/> remote_shell localhost lctl --device 20 destroy 444 1<br/> error: destroy: invalid objid '444'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;<br/> remote_shell localhost lctl --device 21 destroy 444 1<br/> error: destroy: invalid objid '444'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;</p> <p> </p> <p>Any ideas what could be causing this?</p> LU-10663

obdfilter-survey

Bug Major Resolved Fixed John Hammond Mahmoud Hanafi Wed, 14 Feb 2018 00:44:34 +0000 Mon, 12 Mar 2018 21:50:36 +0000 Tue, 27 Feb 2018 04:31:30 +0000 Lustre 2.10.3 Lustre 2.11.0 Lustre 2.10.4 0 7 <p>Hello Mahmoud,<br/> did you interrupt the obdfilter_survey script before getting these errors ?<br/> I am asking because I am presently doing some re-work (as pert of <a href="https://jira.whamcloud.com/browse/LU-9730" title="cleanup OST objects that have been leaked during interrupted/failed runs of obdfilter-survey" class="issue-link" data-issue-key="LU-9730">LU-9730</a>) in the obdfilter_survey framework to strengthen it particularly in its auto-cleanup duty upon normal and interrupted cases.</p> <p>Can you also detail the command line/parameters you have used and also the configuration (single node setup?, direct run on OSS ?, ...) being used ?</p> <p>Did you also check of any "/tmp//obdfilter_survey_*" left files ?</p> <p>This start when I updated from 2.10.1 to 2.10.3. I had ran obdfilter-survey both local(oss) and netdisk(2 osses) and I had interrupted it before in 2.10.1.</p> <p>I checked the file in /tmp didn't find anything.</p> <p>cmd line</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">rszlo=1024 rszhi=4096 size=5000 obdfilter-survey </pre> </div></div> <p>When run obdfilter-survey on a new OST running 2.10.3 the test only runs 1 time. I coped obdfilter-survey and iokit-libecho from a 2.10.1 server still had the issue.</p> <p>I  downgraded the OSS back to 2.10.1 and obdfilter_survey runs without errors.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp7-mds2 ~ # rszlo=1024 rszhi=4096 size=10000 tests_str=<span class="code-quote">"write"</span> obdfilter-survey Wed Feb 14 09:32:41 PST 2018 Obdfilter-survey <span class="code-keyword">for</span> <span class="code-keyword">case</span>=disk from nbp7-mds2 ost 3 sz 30720000K rsz 1024K obj 3 thr 3 write 1096.05 [ 335.99, 448.97] ost 3 sz 30720000K rsz 1024K obj 3 thr 6 write 1163.57 [ 377.99, 754.98] ost 3 sz 30720000K rsz 1024K obj 3 thr 12 write 1163.78 [ 383.00, 776.98] ost 3 sz 30720000K rsz 1024K obj 3 thr 24 write 1164.07 [ 385.99, 778.96] ost 3 sz 30720000K rsz 1024K obj 3 thr 48 write 1164.01 [ 384.99, 777.39]  .... </pre> </div></div> <p>So the bug is in 2.10.3 lctl!</p> <p>Can I get the priority of this case increased 1 level.**</p> <p>copied lctl from a 2.10.1 server to a 2.10.3 obdfilter-survey works. Attaching debug output</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31305" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31305</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10663" title="obdfilter-survey" class="issue-link" data-issue-key="LU-10663"><del>LU-10663</del></a> utils: clear errno before check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 70e9e7bcf4505cba8853117e2eeb92a01e399eec</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31305/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31305/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10663" title="obdfilter-survey" class="issue-link" data-issue-key="LU-10663"><del>LU-10663</del></a> utils: clear errno before check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e488fe9413184e61dcf405c9c87ca348dd6824a</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31430" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31430</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10663" title="obdfilter-survey" class="issue-link" data-issue-key="LU-10663"><del>LU-10663</del></a> utils: clear errno before check<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 800da3bd685aa2ad4c4ed730ac86d79f7693bfc1</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31430/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31430/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10663" title="obdfilter-survey" class="issue-link" data-issue-key="LU-10663"><del>LU-10663</del></a> utils: clear errno before check<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: d140cab6f9bbef3d7f77b91628fe8202517fa185</p> Related LU-10566 Development Rank 1|hzzstb: Rank (Obsolete) 9223372036854775807 Severity