https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8287 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/c7f684c6-2a8e-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c7f684c6-2a8e-11e6-acf3-5254006e85c2</a>.</p> <p>This has failed quite a number of times, but was overshadowed by <a href="https://jira.whamcloud.com/browse/LU-8279" title="sanity-scrub test_4b: @@@@@@ FAIL: Error in dmesg detected" class="issue-link" data-issue-key="LU-8279"><del>LU-8279</del></a> also causing a large number of sanity-sec failures.</p> <p>The sub-test test_16 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mgs and c0 idmap mismatch, 10 attempts </pre> </div></div> <p>It looks like the OST rebooted for some reason, since the modules are not loaded and the dmesg log is empty:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-54vm8: opening /dev/lnet failed: No such device trevis-54vm8: hint: the kernel modules may not be loaded trevis-54vm8: IOC_LIBCFS_GET_NI error 19: No such device /usr/lib64/lustre/tests/sanity-sec.sh: line 981: [: ==: unary operator expected </pre> </div></div> <p>Info required for matching: sanity-sec 16</p> LU-8287

sanity-sec test_16: mgs and c0 idmap mismatch, 10 attempts

Bug Blocker Resolved Fixed Kit Westneat Maloo Wed, 15 Jun 2016 22:08:09 +0000 Fri, 18 Nov 2016 15:50:59 +0000 Fri, 11 Nov 2016 12:54:48 +0000 Lustre 2.9.0 0 9 <p>This might be caused by a failure during test_15:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-19vm4: error: get_param: param_path 'nodemap/25503_3/id': No such file or directory CMD: trevis-19vm4 /usr/sbin/lctl nodemap_del 25503_4 CMD: trevis-19vm4 /usr/sbin/lctl get_param nodemap.25503_4.id </pre> </div></div> <p>Unfortunately, I can't find console logs for any of these failures.</p> <p>This is now the leading cause of autotest patch review test failures.</p> <p>There seem to be two different problems, although they may be related. The first is kernel panics on ZFS OSSes, and the second is a problem syncing the config with a second MDS.</p> <p>zfs example - <a href="https://testing.hpdd.intel.com/test_sets/c7f684c6-2a8e-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c7f684c6-2a8e-11e6-acf3-5254006e85c2</a><br/> ldiskfs example - <a href="https://testing.hpdd.intel.com/test_sets/0daa6758-3557-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0daa6758-3557-11e6-acf3-5254006e85c2</a></p> <p>I wonder if there is some issue with deleting and recreating the index file on ZFS that is causing OSS to kernel panic. It looks like sanity-sec hasn't passed on a ZFS system in a long time. Do you know why there is no console log on these test cases? I thought that there used to be a console log on all tests. It's going to be difficult to figure out what is going on without the kernel panic trace. On the other hand, if the maloo cases are any indication, it should be easy to reproduce.</p> <p>For the second problem, there isn't enough info to figure out what is going on. I'll try to reproduce it locally, and write a debug patch to get more information. Is there a list somewhere of the default maloo debug flags?</p> <p>Thanks,<br/> Kit</p> <p>I've filed an internal ticket regarding the console logs. </p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/20896" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20896</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8287" title="sanity-sec test_16: mgs and c0 idmap mismatch, 10 attempts" class="issue-link" data-issue-key="LU-8287"><del>LU-8287</del></a> nodemap: add more debug messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4c46cdf8acf877f6a4a617099be397013fe6902</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/20954" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20954</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8287" title="sanity-sec test_16: mgs and c0 idmap mismatch, 10 attempts" class="issue-link" data-issue-key="LU-8287"><del>LU-8287</del></a> nodemap: don't stop config lock when target stops<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 59939efdad721ef3e48a7104e020b627383ffa88</p> <p>More failure instance on master branch: <a href="https://testing.hpdd.intel.com/test_sets/d6c0079e-403b-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d6c0079e-403b-11e6-acf3-5254006e85c2</a></p> <p>This is affecting patch review testing on master branch.</p> <p>One more failure instance on master branch: <a href="https://testing.hpdd.intel.com/test_sets/6b55672c-4fa6-11e6-bf87-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6b55672c-4fa6-11e6-bf87-5254006e85c2</a></p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/aa77a6ac-53c2-11e6-a39e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/aa77a6ac-53c2-11e6-a39e-5254006e85c2</a></p> <p>One more failure instance on master: <a href="https://testing.hpdd.intel.com/test_sets/7d5b308e-594f-11e6-b5b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7d5b308e-594f-11e6-b5b1-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20954/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20954/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8287" title="sanity-sec test_16: mgs and c0 idmap mismatch, 10 attempts" class="issue-link" data-issue-key="LU-8287"><del>LU-8287</del></a> nodemap: don't stop config lock when target stops<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 45ce1044cd7b94621e1161cd23c600f8e1c18317</p> <p>Landed for 2.9</p> <p>This issue is still seen on master, build# 3468 for Full EL7.2 Server/EL7.2 Client - ZFS causing following tests to fail.<br/> <a href="https://testing.hpdd.intel.com/test_sets/e2e0de84-a258-11e6-bf05-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e2e0de84-a258-11e6-bf05-5254006e85c2</a></p> <p>The root cause in that case is due to the OSS kernel panic'ing, you can see that the Lustre modules are no longer loaded:<br/> onyx-35vm4: opening /dev/lnet failed: No such device<br/> onyx-35vm4: hint: the kernel modules may not be loaded<br/> onyx-35vm4: IOC_LIBCFS_GET_NI error 19: No such device</p> <p>Unfortunately the console logs don't appear to have been saved, is there a way to make sure these are saved?</p> <p>Reopening as this issue is still seen on master.<br/> <a href="https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2</a></p> <p>For each ZFS run of this test, it looks as though the OST vm has been rebooted in a previous test run because all the dmesg's are from a clean system, that hasn't had any tests from autotest run on it at all (no lctl mark statements).</p> <p>All ZFS in last 4 weeks (all failures):<br/> <a href="https://testing.hpdd.intel.com/test_sets/d0448e1a-93df-11e6-91aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d0448e1a-93df-11e6-91aa-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> (uncaught)<br/> <a href="https://testing.hpdd.intel.com/test_sets/969f1d60-9a60-11e6-a546-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/969f1d60-9a60-11e6-a546-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> (uncaught)<br/> <a href="https://testing.hpdd.intel.com/test_sets/b06c944a-9a63-11e6-a5e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b06c944a-9a63-11e6-a5e5-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b3a81182-9f06-11e6-a747-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b3a81182-9f06-11e6-a747-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> (uncaught)<br/> <a href="https://testing.hpdd.intel.com/test_sets/e2e0de84-a258-11e6-bf05-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e2e0de84-a258-11e6-bf05-5254006e85c2</a> - no OST logs for test_9<br/> <a href="https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a><br/> <a href="https://testing.hpdd.intel.com/test_sets/6196c4ec-a63b-11e6-bf77-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6196c4ec-a63b-11e6-bf77-5254006e85c2</a> &lt;-- <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> (uncaught)</p> <p><a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> is an LBUG in test 9 that doesn't cause a TIMEOUT, just marked as a test failure. (possibility that same LBUG is un-caught in other cases?)</p> <p>While testing patch <a href="http://review.whamcloud.com/23778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23778</a> for <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> on master branch, sanity-sec test 16 failed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>nodemap.c0.idmap= [ { idtype: uid, client_id: 60003, fs_id: 60000 }, { idtype: uid, client_id: 60004, fs_id: 60002 }, { idtype: gid, client_id: 60003, fs_id: 60000 }, { idtype: gid, client_id: 60004, fs_id: 60002 } ] OTHER - IP: 10.2.4.167 sanity-sec test_16: @@@@@@ FAIL: mgs and c0 idmap mismatch, 10 attempts </pre> </div></div> <p>Is this a regression from that patch or the failure was not resolved on master branch? </p> <p>Hi Jian,</p> <p>Patch 23778 fixes an error handling bug, but the original error on ZFS still exists. The original nodemap on ZFS bug that caused this test failure is fixed in change 23849:<br/> <a href="http://review.whamcloud.com/#/c/23849/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/23849/</a></p> Related LU-3291 Development Rank 1|hzyetb: Rank (Obsolete) 9223372036854775807 Severity