https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7776 Lustre <p>lnet-selftest test fails in test setup</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> stdout.log 1 UP mgs MGS MGS 5 2 UP mgc MGC192.168.108.18@tcp c0ab2420-8f51-ad18-f779-591cad596879 5 3 UP mds MDS MDS_uuid 3 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 8 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 9 UP osd-ldiskfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 10 UP ost OSS OSS_uuid 3 11 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 7 12 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 13 UP osd-ldiskfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 14 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 7 15 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 21 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 22 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 Modules still loaded: lustre/osp/osp.o lustre/lod/lod.o lustre/ost/ost.o lustre/mdt/mdt.o lustre/mdd/mdd.o lustre/mgs/mgs.o ldiskfs/ldiskfs.o lustre/quota/lquota.o lustre/lfsck/lfsck.o lustre/mgc/mgc.o lustre/fid/fid.o lustre/fld/fld.o lustre/ptlrpc/ptlrpc.o lustre/obdclass/obdclass.o lnet/klnds/socklnd/ksocklnd.o lnet/lnet/lnet.o libcfs/libcfs/libcfs.o </pre> </div></div> Solo setup LU-7776

lustre-single lnet-selftest test failed

Bug Minor Resolved Fixed WC Triage Abrar-ahmed Mon, 15 Feb 2016 08:57:42 +0000 Wed, 15 Jun 2016 12:53:55 +0000 Wed, 15 Jun 2016 12:53:55 +0000 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.9.0 0 6 <p>lnet-selftest.sh script is fails while trying to execute cleanupall() during test setup. cleanupall() in turn fails trying to remove modules while still in use. This happens on a solo setup when local_node returns true and variable CLIENTONLY is set to true. Further cleanupall() internally calls stopall() which checks CLIENTONLY and returns midway if true without further cleanup of mgs, mds and ost. This causes cleanupall() to fail at a later stage trying to remove loaded modules.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>stopall() { ... [ "$CLIENTONLY" ] &amp;&amp; return </pre> </div></div> <p>History of change shows that this regression was introduced as a result of a debug patch <a href="http://review.whamcloud.com/12469" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12469</a> (<a href="https://jira.whamcloud.com/browse/LU-4181" title="lnet-selftest test_smoke: lst error found" class="issue-link" data-issue-key="LU-4181"><del>LU-4181</del></a> tests: cleanup lustre before starting lnet-selftest.sh)<br/> As the discussions on <a href="https://jira.whamcloud.com/browse/LU-4181" title="lnet-selftest test_smoke: lst error found" class="issue-link" data-issue-key="LU-4181"><del>LU-4181</del></a> point out that the changes were for debug purpose and removing the modules was not a necessity i propose revoking this change to resolve the bug.</p> <p>I don't think that reverting the patch is a good idea, since I believe this will cause lnet-selftest to begin failing again in our test configuration.</p> <p>Instead, I think it should be enough to change the "cleanupall" to "stopall" so that it doesn't try to unload the modules, which isn't necessary. The goal of the <a href="https://jira.whamcloud.com/browse/LU-4181" title="lnet-selftest test_smoke: lst error found" class="issue-link" data-issue-key="LU-4181"><del>LU-4181</del></a> patch was to stop the clients so that they would not interfere with the testing, or become disconnected when lnet-selftest was saturating the network.</p> <p>@Andreas Dilger</p> <p>Here is my understanding of the debug patch submitted via commit &lt;a8ba5c645f91faf86a84c99dd2cc049bc54e12b1&gt;<br/> Debug patch replaced stopall with cleanupall. cleanupall in addition to unmounting clients and stopping servers also unloads modules which i believe was the intended purpose of the debug patch. Please correct my understanding if wrong.<br/> Quoting the relevant section of the debug patch change below</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">- local_mode &amp;&amp; CLIENTONLY=yes - stopall - RESTORE_MOUNT=yes + local_mode &amp;&amp; CLIENTONLY=yes + RESTORE_MOUNT=yes + LOAD_MODULES_REMOTE=<span class="code-keyword">true</span> + cleanupall </pre> </div></div> <p>So changing cleanupall to stopall would be functionally reverting the debug patch. Would this not cause your test setup to fail again?.</p> <p>@Andreas Dilger</p> <p>Alternate solution to keep the debug patch functionality would be to avoid calling cleanupall on local_mode setups. Something like below</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">- local_mode &amp;&amp; CLIENTONLY=yes + <span class="code-keyword">if</span> local_mode; then + CLIENTONLY=yes + stopall + <span class="code-keyword">else</span> + LOAD_MODULES_REMOTE=<span class="code-keyword">true</span> + cleanupall + fi </pre> </div></div> <p>Let me know which solution works for you or if you want to suggest alternatives. I can upload a patch for the same.</p> <p>Looks reasonable, and if this patch works for you then you can submit it and it can be tested.</p> <p>Abrarahmed Momin (kais_abrar@yahoo.co.in) uploaded a new patch: <a href="http://review.whamcloud.com/19308" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19308</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7776" title="lustre-single lnet-selftest test failed" class="issue-link" data-issue-key="LU-7776"><del>LU-7776</del></a> tests: lnet-selftest.sh local_mode failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4efad14f8c85d5346aed97a047d9e5681c1792e5</p> <p>@Andreas Dilge: Hi Andreas, have uploaded the discussed patch and test run results were fine. Can you and others kindly review the patch.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19308/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19308/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7776" title="lustre-single lnet-selftest test failed" class="issue-link" data-issue-key="LU-7776"><del>LU-7776</del></a> tests: lnet-selftest.sh local_mode failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 84030bf26c1763edf9ac17a8cd2765e9163294bf</p> <p>patch has landed to master for 2.9.0</p> Development Epic/Theme test Rank 1|hzy1an: Rank (Obsolete) 9223372036854775807 Severity