https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15706 Lustre <p>"lctl pool_destroy" can't work correctly when MGS and MDT0 is from different node.</p> <p>Then I did the following test on two nodes(centos7-2 and centos7-4) and found it failed because it didn't process "SKIP" pool llog records properly.</p> <p>centos7-2 (MDT+2OST+client):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos7-2 tests]# MGSDEV=/tmp/lustre-mgs mgs_HOST=centos7-4 mdt_HOST=centos7-2 ost1_HOST=centos7-2 ost2_HOST=centos7-2 PDSH="pdsh -S -Rssh -w" sh llmount.sh </pre> </div></div> <p>centos7-4(MGS):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos7-4 tests]# lctl pool_new lustre.testpool Pool lustre.testpool created [root@centos7-4 tests]# lctl pool_add lustre.testpool OST0000 OST lustre-OST0000_UUID added to pool lustre.testpool [root@centos7-4 tests]# lctl pool_add lustre.testpool OST0001 OST lustre-OST0001_UUID added to pool lustre.testpool [root@centos7-4 tests]# lctl pool_destroy lustre.testpool Pool lustre.testpool not empty, please remove all members pool_destroy: Directory not empty [root@centos7-4 tests]# lctl pool_remove lustre.testpool OST0000 OST lustre-OST0000_UUID removed from pool lustre.testpool [root@centos7-4 tests]# lctl pool_destroy lustre.testpool Pool lustre.testpool destroyed </pre> </div></div> <p>"testpool" was destroyed wrongly even OST0001 was still in the pool.</p> <p>After checking the llog records, I found "SKIP pool add 0:lustre-clilov  1:lustre  2:testpool  3:lustre-OST0000_UUID" was skipped.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#43 (112)SKIP pool new 0:lustre-clilov 1:lustre 2:testpool #44 (224)SKIP END marker 29 (flags=0x06, v2.15.0.0) lustre-clilov 'new lustre.testpool' Wed Mar 30 11:02:17 2022-Wed Mar 30 11:02:51 2022 #45 (224)SKIP START marker 31 (flags=0x05, v2.15.0.0) lustre-clilov 'add lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:28 2022-Wed Mar 30 11:02:49 2022 #46 (136)SKIP pool add 0:lustre-clilov 1:lustre 2:testpool 3:lustre-OST0000_UUID #48 (224)SKIP END marker 31 (flags=0x06, v2.15.0.0) lustre-clilov 'add lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:28 2022-Wed Mar 30 11:02:49 2022 #49 (224)marker 33 (flags=0x01, v2.15.0.0) lustre-clilov 'add lustre.testpool.lustre-OST0001_UUID' Wed Mar 30 11:02:30 2022- #50 (136)pool add 0:lustre-clilov 1:lustre 2:testpool 3:lustre-OST0001_UUID #51 (224)END marker 33 (flags=0x02, v2.15.0.0) lustre-clilov 'add lustre.testpool.lustre-OST0001_UUID' Wed Mar 30 11:02:30 2022- #52 (224)marker 35 (flags=0x01, v2.15.0.0) lustre-clilov 'rem lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:49 2022- #53 (136)pool remove 0:lustre-clilov 1:lustre 2:testpool 3:lustre-OST0000_UUID #54 (224)END marker 35 (flags=0x02, v2.15.0.0) lustre-clilov 'rem lustre.testpool.lustre-OST0000_UUID' Wed Mar 30 11:02:49 2022- #55 (224)marker 37 (flags=0x01, v2.15.0.0) lustre-clilov 'del lustre.testpool' Wed Mar 30 11:02:51 2022- #56 (112)pool destroy 0:lustre-clilov 1:lustre 2:testpool #57 (224)END marker 37 (flags=0x02, v2.15.0.0) lustre-clilov 'del lustre.testpool' Wed Mar 30 11:02:51 2022- </pre> </div></div> <p>I will look into this issue.</p> LU-15706

"lctl pool_destroy" can't work correctly due to "SKIP" records

Bug Minor Resolved Fixed Emoly Liu Emoly Liu Wed, 30 Mar 2022 07:45:05 +0000 Fri, 20 Oct 2023 11:49:42 +0000 Mon, 27 Jun 2022 15:52:36 +0000 Lustre 2.16.0 Lustre 2.15.2 0 5 <p>"Emoly Liu &lt;emoly@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46951" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46951</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15706" title="&quot;lctl pool_destroy&quot; can&#39;t work correctly due to &quot;SKIP&quot; records" class="issue-link" data-issue-key="LU-15706"><del>LU-15706</del></a> lctl: deal with "SKIP" pool record correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 67aca5483a7718c75b82b9fb20c3ad1fccd4b700</p> <p>During Maloo test and my local test, I also noticed the following issue, that is, if jt_llog_print_iter() tries to get llog records more times in for loop, the "boundary" record is easy to be processed by llog_print_cb()-&gt;class_config_yaml_output() improperly, skipped or not skipped wrongly. </p> <p>For example, in the failure at <a href="https://testing.whamcloud.com/test_logs/ca4871c9-d298-4a2a-99f6-7449aa6fb803/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/ca4871c9-d298-4a2a-99f6-7449aa6fb803/show_text</a> , it says "trevis-39vm5: OST lustre-OST0006_UUID is not part of the 'lustre' fs.". I can reproduce this issue locally, and I found the reason as follows:<br/> The original llog records output by debugfs are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#23 (128)attach 0:lustre-OST0000-osc 1:osc 2:lustre-clilov_UUID #24 (144)setup 0:lustre-OST0000-osc 1:lustre-OST0000_UUID 2:192.168.0.118@tcp #25 (128)lov_modify_tgts add 0:lustre-clilov 1:lustre-OST0000_UUID 2:0 3:1 </pre> </div></div> <p>But, the record output by class_config_yaml_output() shows 24th and 25th records were skipped wrongly, that is why OST0000 can't be found.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- { index: 22, event: add_uuid, nid: 192.168.0.118@tcp(0x20000c0a80076), node: 192.168.0.118@tcp } - { index: 23, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } ======BOUNDARY====== - { index: 31, event: add_uuid, nid: 192.168.0.118@tcp(0x20000c0a80076), node: 192.168.0.118@tcp } - { index: 32, event: attach, device: lustre-OST0001-osc, type: osc, UUID: lustre-clilov_UUID } </pre> </div></div> <p> </p> <p>Another, in lost-pool.sh test23a failed due to "Pool lustre.testpool not empty, please remove all members pool_destroy: Directory not empty ", because the 1412th record was not skipped correctly.</p> <p>The original llog records output by debugfs are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#1412 (136)SKIP pool add 0:lustre-clilov 1:lustre 2:testpool 3:lustre-OST0003_UUID #1413 (224)SKIP END marker 939 (flags=0x06, v2.15.0.0) lustre-clilov 'add lustre.testpool.lustre-OST0003_UUID' Fri Apr 1 08:47:01 2022-Fri Apr 1 08:47:02 2022 </pre> </div></div> <p>But class_config_yaml_output() didn't add SKIP to this record properly, so that OST0003 was considered still in testpool wrongly.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>=====BOUNDARY===== - { index: 1412, event: add_pool, device: lustre-clilov, fsname: lustre, pool: testpool, ost: lustre-OST0003_UUID } - { index: 1428, event: remove_pool, device: lustre-clilov, fsname: lustre, pool: testpool, ost: lustre-OST0001_UUID } </pre> </div></div> <p> </p> <p>I will look into the following part of code in class_config_yaml_output() and make a fix.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (lcfg-&gt;lcfg_command == LCFG_MARKER) { struct cfg_marker *marker = lustre_cfg_buf(lcfg, 1); lustre_swab_cfg_marker(marker, swab, LUSTRE_CFG_BUFLEN(lcfg, 1)); if (marker-&gt;cm_flags &amp; CM_START) { *cfg_flags = CFG_F_MARKER; if (marker-&gt;cm_flags &amp; CM_SKIP) *cfg_flags = CFG_F_SKIP; } else if (marker-&gt;cm_flags &amp; CM_END) { *cfg_flags = 0; } if (likely(!raw)) return 0; } /* entries outside marker are skipped */ if (!(*cfg_flags &amp; CFG_F_MARKER) &amp;&amp; !raw) return 0; /* inside skipped marker */ if (*cfg_flags &amp; CFG_F_SKIP &amp;&amp; !raw) return 0; </pre> </div></div> <p>The root cause of this issue is that if the "start" several record in one ioctl request are just the ones "protected" by one marker, they will don't know their "SKIP or not" flag, and then will be mis-labeled in class_config_yaml_output().</p> <p>I'm trying to remember the flag in llog_print_cb() to fix this issue.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46951/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46951/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15706" title="&quot;lctl pool_destroy&quot; can&#39;t work correctly due to &quot;SKIP&quot; records" class="issue-link" data-issue-key="LU-15706"><del>LU-15706</del></a> llog: deal with "SKIP" pool llog records correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 474f2670d63b66a77ee3acb72b18bc7b5afbec84</p> <p>Landed for 2.16</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49230" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49230</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15706" title="&quot;lctl pool_destroy&quot; can&#39;t work correctly due to &quot;SKIP&quot; records" class="issue-link" data-issue-key="LU-15706"><del>LU-15706</del></a> llog: deal with "SKIP" pool llog records correctly<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 8f3127f0e20992a630838db5f65daf4b008c5dc3</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49230/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49230/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15706" title="&quot;lctl pool_destroy&quot; can&#39;t work correctly due to &quot;SKIP&quot; records" class="issue-link" data-issue-key="LU-15706"><del>LU-15706</del></a> llog: deal with "SKIP" pool llog records correctly<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 70accf10fac5032e185d981b0a0193aad9980371</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52773" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52773</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15706" title="&quot;lctl pool_destroy&quot; can&#39;t work correctly due to &quot;SKIP&quot; records" class="issue-link" data-issue-key="LU-15706"><del>LU-15706</del></a> llog: deal with "SKIP" pool llog records correctly<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 30a7ee61140655d2dac1fd43ade538a71b384787</p> Related LU-15142 LU-16324 Development Rank 1|i02m13: Rank (Obsolete) 9223372036854775807 Severity