https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8824 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/abdb13dc-a627-11e6-964e-5254006e85c2</a>.</p> <p>The sub-test test_9 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-34vm4:LBUG/LASSERT detected </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>02:04:11:[15785.994958] Lustre: DEBUG MARKER: == sanity-sec test 9: nodemap range add ============================================================== 02:02:49 (1478656969) 02:04:11:[15792.826885] Lustre: 10421:0:(nodemap_handler.c:1020:nodemap_create()) adding nodemap '27295_7' to config without default nodemap 02:04:11:[15792.830823] Lustre: 10421:0:(nodemap_handler.c:1020:nodemap_create()) Skipped 3 previous similar messages 02:04:11:[15800.705743] Lustre: 10421:0:(mgc_request.c:1756:mgc_process_recover_nodemap_log()) MGC10.9.5.176@tcp: error processing nodemap log nodemap: rc = -2 02:04:11:[15800.709914] LustreError: 10421:0:(nodemap_handler.c:1428:nodemap_config_set_active()) ASSERTION( config-&gt;nmc_default_nodemap ) failed: 02:04:11:[15800.714076] LustreError: 10421:0:(nodemap_handler.c:1428:nodemap_config_set_active()) LBUG 02:04:11:[15800.716317] Pid: 10421, comm: ll_cfg_requeue 02:04:11:[15800.718308] 02:04:11:[15800.718308] Call Trace: 02:04:11:[15800.721741] [&lt;ffffffffa09387d3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] 02:04:11:[15800.723818] [&lt;ffffffffa0938d75&gt;] lbug_with_loc+0x45/0xc0 [libcfs] 02:04:11:[15800.725837] [&lt;ffffffffa0d34a17&gt;] nodemap_config_set_active+0x2a7/0x2e0 [ptlrpc] 02:04:11:[15800.727873] [&lt;ffffffffa0d3d908&gt;] nodemap_config_set_active_mgc+0x38/0x1e0 [ptlrpc] 02:04:11:[15800.729985] [&lt;ffffffffa0ca28f0&gt;] ? ptlrpc_request_cache_free+0x90/0x1d0 [ptlrpc] 02:04:11:[15800.732071] [&lt;ffffffffa0ca35d5&gt;] ? __ptlrpc_req_finished+0x475/0x690 [ptlrpc] 02:04:11:[15800.734162] [&lt;ffffffffa0c43e6b&gt;] mgc_process_recover_nodemap_log+0x34b/0xe10 [mgc] 02:04:11:[15800.736195] [&lt;ffffffffa0c46894&gt;] mgc_process_log+0x754/0x880 [mgc] 02:04:11:[15800.738132] [&lt;ffffffff816399cd&gt;] ? schedule_timeout+0x17d/0x2d0 02:04:11:[15800.740126] [&lt;ffffffffa09439d7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] 02:04:11:[15800.742013] [&lt;ffffffffa0c48908&gt;] mgc_requeue_thread+0x2b8/0x880 [mgc] 02:04:11:[15800.744113] [&lt;ffffffff810b8940&gt;] ? default_wake_function+0x0/0x20 02:04:11:[15800.746313] [&lt;ffffffffa0c48650&gt;] ? mgc_requeue_thread+0x0/0x880 [mgc] 02:04:11:[15800.748437] [&lt;ffffffff810a5b8f&gt;] kthread+0xcf/0xe0 02:04:11:[15800.750331] [&lt;ffffffff810a5ac0&gt;] ? kthread+0x0/0xe0 02:04:11:[15800.752203] [&lt;ffffffff81646c98&gt;] ret_from_fork+0x58/0x90 02:04:11:[15800.754097] [&lt;ffffffff810a5ac0&gt;] ? kthread+0x0/0xe0 02:04:11:[15800.755897] </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity-sec 9</p> LU-8824

sanity-sec test_9: ASSERTION( config->nmc_default_nodemap )

Bug Blocker Resolved Fixed Kit Westneat Maloo Thu, 10 Nov 2016 19:37:34 +0000 Tue, 6 Jul 2021 19:01:40 +0000 Wed, 23 Nov 2016 04:50:52 +0000 Lustre 2.9.0 0 6 <p><a href="https://testing.hpdd.intel.com/test_sets/b06c944a-9a63-11e6-a5e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b06c944a-9a63-11e6-a5e5-5254006e85c2</a></p> <p>Uncaught failures in past 4 weeks (console log of OST vm shows LBUG):<br/> <a href="https://testing.hpdd.intel.com/test_sets/969f1d60-9a60-11e6-a546-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/969f1d60-9a60-11e6-a546-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/d0606342-93df-11e6-91aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/d0606342-93df-11e6-91aa-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/ff975cc6-9ed7-11e6-b8c4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/ff975cc6-9ed7-11e6-b8c4-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/b3b42986-9f06-11e6-a747-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/b3b42986-9f06-11e6-a747-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/619feb44-a63b-11e6-bf77-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/619feb44-a63b-11e6-bf77-5254006e85c2</a></p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23706" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23706</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> test: EXCEPT test_9 till fixed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a469c81df1a8fea2798c8fe7866456c53a53a00d</p> <p>Kit</p> <p>Could you please advise on this issue and how whether we could live with this in 2.9?</p> <p>Peter</p> <p>Thanks for gathering the logs on this Nathaniel.</p> <p>It looks like there is an error handling issue in mgc_process_recover_nodemap_log. I can fix up the error handling for the nodemap portion, but the handling for the recovery log part is also missing, and I don't feel confident adding it there. Is there someone who can look at that portion?</p> <p>I'm not sure what the root cause is yet, though it looks like something to do with the default nodemap not getting transfered correctly - does this LBUG happen on all ZFS full group tests or is it more random? </p> <p>Fixing up the error handling should be enough for 2.9, though it means that nodemap will be only partially functional on ZFS systems. </p> <p>It seems to happen anytime sanity-sec is run on ZFS, but not when run on ldiskfs.</p> <p>I think I've figured out what's going on. The config load code expects the index file to return the key/values in key-sorted order, which the ldiskfs index files do. The ZFS index files however appear to return the keys in hash sorted order, at least according to this comment:<br/> /*</p> <ul> <li>XXX: implement support for fixed-size keys sorted with natural</li> <li>numerical way (not using internal hash value)<br/> */</li> </ul> <p>We currently embed the config record type in the key so that create records are processed before update records, and so not having the records sent in key-order breaks this. </p> <p>I'm going to investigate how easy it would be to modify the config load/send operation to have it do a two-pass load, where the create records would be loaded first, and then the other records could be loaded after. </p> <p>Kit,</p> <p>Awesome find.</p> <p>EXCEPTing test_9 just delays the ASSERTION to test_15:<br/> <a href="https://testing.hpdd.intel.com/sub_tests/aaedadbe-a888-11e6-b6bd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/aaedadbe-a888-11e6-b6bd-5254006e85c2</a></p> <p>I'm think getting a real fix is necissary for sanity-sec to pass with ZFS.</p> <p>Kit</p> <p>This is indeed good news. How are things progressing on making the changes necessary with the error handling?</p> <p>Peter</p> <p>Hi Peter,</p> <p>I can get a patch up for the error handling tonight or tomorrow. Fixing the config loading and unloading will take a bit longer, but I'll ty to get a patch up by the end of the week.</p> <ul class="alternate" type="square"> <li>Kit</li> </ul> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/23778" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23778</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> nodemap: properly handle errors loading nodemap conf<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0ae8e3db5cd16acc4f3bde47a896b05a01383c9b</p> <p>Thanks Kit! This is encouraging news</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/23849" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23849</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> nodemap: load nodemap definitions first<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7d5800455161e0d2fca47a1754b7fc734d4a2999</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23778/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23778/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> nodemap: properly handle errors loading nodemap conf<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9be888d56caf73184f72a4ad782196d255331ee2</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23849/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23849/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8824" title="sanity-sec test_9: ASSERTION( config-&gt;nmc_default_nodemap )" class="issue-link" data-issue-key="LU-8824"><del>LU-8824</del></a> nodemap: load nodemap definitions first<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 89ce9d5b125762f39339916f14c01242107739ed</p> <p>Landed for 2.9</p> Duplicate LU-8825 LU-8850 Related Development Rank 1|hzyv7r: Rank (Obsolete) 9223372036854775807 Severity