https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8305 Lustre <p>This issue was created by maloo for parinay &lt;parinay_kondekar@xyratex.com&gt;</p> <p>Please provide additional information about the failure here.</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/aba1b6c8-34f8-11e6-80b9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/aba1b6c8-34f8-11e6-80b9-5254006e85c2</a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4789:error() = /usr/lib64/lustre/tests/sanity-sec.sh:1784:test_27() = /usr/lib64/lustre/tests/test-framework.sh:5054:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5093:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4939:run_test() = /usr/lib64/lustre/tests/sanity-sec.sh:1789:main() Dumping lctl log to /logdir/test_logs/2016-06-17/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__39912__-69959014310400-174851/sanity-sec.test_27.*.1466214942.log CMD: trevis-55vm1.trevis.hpdd.intel.com,trevis-55vm2,trevis-55vm3,trevis-55vm7,trevis-55vm8 /usr/sbin/lctl dk &gt; /logdir/test_logs/2016-06-17/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__39912__-69959014310400-174851/sanity-sec.test_27.debug_log.\$(hostname -s).1466214942.log; dmesg &gt; /logdir/test_logs/2016-06-17/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__39912__-69959014310400-174851/sanity-sec.test_27.dmesg.\$(hostname -s).1466214942.log CMD: trevis-55vm3,trevis-55vm7,trevis-55vm8 /usr/sbin/lctl set_param debug=\"\" trevis-55vm3: error: set_param: setting debug: no value trevis-55vm8: error: set_param: setting debug: no value trevis-55vm7: error: set_param: setting debug: no value Resetting fail_loc on all nodes...CMD: trevis-55vm1.trevis.hpdd.intel.com,trevis-55vm2,trevis-55vm3,trevis-55vm7,trevis-55vm8 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> parinay@osh:~/Code/lustre-intel$ git show 5042d4c --pretty=fuller commit 5042d4c8ecad287276e04c52c6a1fee9c9b597a9 Author: Sebastien Buisson &lt;sbuisson@ddn.com&gt; AuthorDate: Sat Apr 30 18:27:57 2016 +0200 Commit: Oleg Drokin &lt;oleg.drokin@intel.com&gt; CommitDate: Thu Jun 16 22:15:58 2016 +0000 LU-7846 tests: test for nodemap fileset in sanity-sec Add new tests in sanity-sec.sh to test for fileset in nodemap </pre> </div></div> LU-8305

sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0

Bug Minor Closed Fixed Sebastien Buisson Maloo Mon, 20 Jun 2016 04:21:33 +0000 Thu, 20 Apr 2017 21:35:18 +0000 Wed, 12 Apr 2017 23:19:45 +0000 Lustre 2.10.0 Lustre 2.9.0 Lustre 2.10.0 0 9 <p>The test was added by the commit of <a href="https://jira.whamcloud.com/browse/LU-7846" title="Impose subdirectory mount via nodemap" class="issue-link" data-issue-key="LU-7846"><del>LU-7846</del></a></p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/20990" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20990</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: add traces for sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 03c4796dd2809e01b236035ffb81d57a66091f22</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20990/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20990/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: add traces for sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: aa9938a26b2fd58afc53bd25020e8940464d1855</p> <p>Landed for 2.9</p> <p>Hit on master again:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fdfba8e0-a45e-11e6-8a63-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fdfba8e0-a45e-11e6-8a63-5254006e85c2</a></p> <p>The patch that was landed only had trace/debug added to santy-sec. This test is still failing and we have several instances with the debug information in the client test_log:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/7f5995f6-a5c1-11e6-bf77-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7f5995f6-a5c1-11e6-bf77-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/fdfba8e0-a45e-11e6-8a63-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fdfba8e0-a45e-11e6-8a63-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e5dee85c-a2cf-11e6-8e31-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e5dee85c-a2cf-11e6-8e31-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/32ccc30a-a0af-11e6-a5bb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32ccc30a-a0af-11e6-a5bb-5254006e85c2</a></p> <p>In the past week or so, It looks like sanity-sec test_27 is failing with this error about once every two days. </p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/23693" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23693</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: strengthen fileset cleanup in sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f23055e66fbbe234c0ad9b00552473bf554690e</p> <p>The problem is that fileset is not properly cleared on MGS side first. I pushed a new patch to improve fileset cleanup on MGS side.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3b325832-a8b8-11e6-8969-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3b325832-a8b8-11e6-8969-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23693/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23693/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: strengthen fileset cleanup in sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4827d4e3a3712ca74dbe276aee4ba3dc6069a78b</p> <p>Landed for 2.9</p> <p>Still hit the problem on master, though the message is slightly different: "On MGS, fileset cannnot be cleared"<br/> <a href="https://testing.hpdd.intel.com/test_sets/30023954-b0b1-11e6-9c4b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/30023954-b0b1-11e6-9c4b-5254006e85c2</a></p> <p>Hi,</p> <p>Thanks to the new messages recently added to sanity-sec, I can see that the error is due to the fact that the command "lctl nodemap_set_fileset --name c0 --fileset ''" fails on the MGS. This is weird as it is a local command which purpose is just to modify the fileset info on a nodemap, and especially weird because the named command does not return any error.</p> <p>I am able to reproduce the problem on my test system, but it does not occur systematically, and the only configuration in which I am able to trigger it is with a combo MGT/MDT and a regular MDT on the same node. I am trying to debug the issue at the moment.</p> <p>Thanks,<br/> Sebastien.</p> <p>The problem with sanity-sec test_27 stems from the way the fileset info is setup on the nodemap entry. If we do 'lctl set_param' and then 'lctl set_param -P', it will set the fileset info twice on MGS side. Under certain circumstances, the propagation of 'lctl set_param -P' can be delayed, and happen in test_27 after we have reset the fileset info with 'lctl nodemap_set_fileset'.<br/> So we fix test_27 by only setting fileset on nodemap with 'set_param -P' command. But we need to wait into wait_nm_sync() for it to be set on MGS as well.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/23936" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23936</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: fix fileset setup in sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8dcfaf64f1d62580c3e8cc3d4b3b64542e02bdd2</p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ae2f0e8a-be33-11e6-9b28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ae2f0e8a-be33-11e6-9b28-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23936/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23936/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8305" title="sanity-sec test_27: @@@@@@ FAIL: fileset not cleared on nodemap c0" class="issue-link" data-issue-key="LU-8305"><del>LU-8305</del></a> tests: fix fileset setup in sanity-sec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 947d81daa1ba218bb027b17061c1c24ccaf85c22</p> <p>This issue has resurfaced in the latest tag (2.9.55, b3550).</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/630a4991-3c6d-4755-b50e-9932f0cf69fb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/630a4991-3c6d-4755-b50e-9932f0cf69fb</a></p> <p>&gt; This issue has resurfaced in the latest tag (2.9.55, b3550).<br/> &gt; <a href="https://testing.hpdd.intel.com/test_sessions/630a4991-3c6d-4755-b50e-9932f0cf69fb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/630a4991-3c6d-4755-b50e-9932f0cf69fb</a></p> <p>Not exactly, as clients in this test are running Lustre 2.9.0. Patch <a href="https://review.whamcloud.com/23936" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23936</a> landed after 2.9.0, so sanity-sec.sh executed in this run does not contain the fix (the purpose of the patch was only to address a test script issue).</p> <p>I think this ticket can be closed again.</p> <p>Thanks,<br/> Sebastien.</p> <p>Do we just need to adjust the test script to skip running in interop mode with older releases?</p> <p>Will open a new ticket to modify the sanity-sec test on 2.9.0 clients.</p> Related Development Rank 1|hzyf0n: Rank (Obsolete) 9223372036854775807 Severity