https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15055 Lustre <p>The <tt>qmt_pool_add_rem()</tt> message started appearing on 2021-08-18 in test output, and there have been a few thousand hits per day:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 5659:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0000_UUID pool pool1: rc = -17 LustreError: 5666:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0001_UUID pool pool2: rc = -17 LustreError: 5674:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0002_UUID pool pool3: rc = -17 </pre> </div></div> <p>Patches landed on that day are listed below (no other patches landed after 2021-08-10 or before 2021-08-25):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git log --oneline --before 2021-08-22 --after 2021-08-16 d8204f903a (tag: v2_14_54, tag: 2.14.54) New tag 2.14.54 5220160648 LU-14093 lutf: fix build with gcc10 a205334da5 LU-14903 doc: update lfs-setdirstripe man page 1313cad7a1 LU-14899 ldiskfs: Add 5.4.136 mainline kernel support c44afcfb72 LU-12815 socklnd: set conns_per_peer based on link speed 6e30cd0844 LU-14871 kernel: kernel update RHEL7.9 [3.10.0-1160.36.2.el7] 14b8276e06 LU-14865 utils: llog_reader.c printf type mismatch aa5d081237 LU-9859 lnet: fold lprocfs_call_handler functionality into lnet_debugfs_* e423a0bd7a LU-14787 libcfs: Proved an abstraction for AS_EXITING 76c71a167b LU-14775 kernel: kernel update SLES12 SP5 [4.12.14-122.74.1] 67752f6db2 LU-14773 tests: skip check_network() on working node 024f9303bc LU-14668 lnet: Lock primary NID logic 684943e2d0 LU-14668 lnet: peer state to lock primary nid 16321de596 LU-14661 obdclass: Add peer/peer NI when processing llog * ac201366ad LU-14661 lnet: Provide kernel API for adding peers 51350e9b73 LU-14531 osd: serialize access to object vs object destroy a5cbe7883d LU-12815 socklnd: allow dynamic setting of conns_per_peer d13d8158e8 LU-14093 mgc: rework mgc_apply_recover_logs() for gcc10 8dd4488a07 LU-6142 tests: remove iam_ut binary 301d76a711 LU-14876 out: don't connect to busy MDS-MDS export * </pre> </div></div> <p>It isn't really clear which of those patches started causing this problem, but the message is being printed in different subtests that create/remove OST pools.</p> <p> </p> <p>The graph shows occurrences by subtest since 2021-08-18, it looks like this happens in any subtest that is adding a pool:<br/> <span class="image-wrap" style=""><a id="40785_thumb" href="https://jira.whamcloud.com/secure/attachment/40785/40785_Screen+Shot+2021-09-29+at+14.03.14.png" title="Screen Shot 2021-09-29 at 14.03.14.png" file-preview-type="image" file-preview-id="40785" file-preview-title="Screen Shot 2021-09-29 at 14.03.14.png"><img src="https://jira.whamcloud.com/secure/thumbnail/40785/_thumb_40785.png" style="border: 0px solid black" role="presentation"/></a></span></p> LU-15055

spurious qmt_pool_add_rem() message when adding OST to pool: -EEXIST

Bug Minor Resolved Fixed Sergey Cheremencev Andreas Dilger Fri, 1 Oct 2021 23:51:19 +0000 Mon, 27 Jun 2022 15:56:43 +0000 Mon, 27 Jun 2022 15:56:43 +0000 Lustre 2.16.0 0 5 <p>Alex</p> <p>Any suggestions on this one?</p> <p>Peter</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sergey" class="user-hover" rel="sergey">sergey</a>, I suspect that this message is actually related to a pool quota change? The other possibility is that there is something bad/wrong happening with how OST pools are configured, and the same OST is being added to a pool multiple times.</p> <p>Also related, the formatting of the error message is broken:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CERROR(<span class="code-quote">"%s: can't %s %s pool %s: rc = %d\n"</span>, add ? <span class="code-quote">"add to"</span> : <span class="code-quote">"remove"</span>, obd-&gt;obd_name, slavename, poolname, rc); </pre> </div></div> <p>resulting in:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0000_UUID pool pool1: rc = -17 </pre> </div></div> <p>but it should really be something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>qmt_pool_add_rem()) scratch-QMT0000: can't add scratch-OST0000_UUID to pool 'pool1': rc = -17 </pre> </div></div> <p>The simple solution would be to quiet this console error message for the <tt>-EEXIST</tt>, but it would be better to understand <b>why</b> this message suddenly started being seen, and fix the root cause (e.g. duplicate config records for OSTs, or an issue in the test script?).</p> <p>The only patches that are even tangentially related to configuration that landed in the target window are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16321de596 LU-14661 obdclass: Add peer/peer NI when processing llog 301d76a711 LU-14876 out: don't connect to busy MDS-MDS export </pre> </div></div> <p>and no patches even come close to the OST pools or quota code or tests in this window. Possibly something in <a href="https://jira.whamcloud.com/browse/LU-14661" title="Provide kernel API for adding peer/peer NI" class="issue-link" data-issue-key="LU-14661"><del>LU-14661</del></a> <a href="https://jira.whamcloud.com/browse/LU-14876" title="OUT: possible concurrent execution of UPDATE request and its resent" class="issue-link" data-issue-key="LU-14876"><del>LU-14876</del></a> is causing the OST to be registered twice? Looking at the MDS logs of a recent test run:<br/> <a href="https://testing.whamcloud.com/test_sets/bb6c5d01-df84-47fd-8bd9-21f08be29658" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/bb6c5d01-df84-47fd-8bd9-21f08be29658</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[11762.531609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 1b: Quota pools: Block hard limit \(normal use and out of quota\) === 14:55:59 \(1637679359\) (1637673981) [11780.265188] Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1 [11788.955306] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2&gt;/dev/null || echo foo [11790.247590] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool1 2&gt;/dev/null || echo foo [11791.573141] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0007/1] [11802.006568] LustreError: 624610:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't lustre-QMT0000 lustre-OST0000_UUID pool qpool1: rc = -17 [11802.008583] LustreError: 624610:0:(qmt_pool.c:1390:qmt_pool_add_rem()) Skipped 1 previous similar message : [11907.682379] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [11908.326534] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1 [11908.975981] Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID [11916.181171] LustreError: 632050:0:(qmt_pool.c:1390:qmt_pool_add_rem()) remove: can't lustre-QMT0000 lustre-OST0000_UUID pool qpool1: rc = -22 [11916.183136] LustreError: 632050:0:(qmt_pool.c:1390:qmt_pool_add_rem()) Skipped 7 previous similar messages [11917.638271] Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID [11926.306964] Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0002_UUID [11935.124558] LustreError: 632331:0:(qmt_pool.c:1390:qmt_pool_add_rem()) remove: can't lustre-QMT0000 lustre-OST0002_UUID pool qpool1: rc = -22 [11935.126564] LustreError: 632331:0:(qmt_pool.c:1390:qmt_pool_add_rem()) Skipped 1 previous similar message [11936.973878] Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0003_UUID [11945.649745] Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0004_UUID </pre> </div></div> <p>so it is also <b>removing</b> the same OST from the pool twice and <tt>lu_tgt_pool_remove()</tt> is returning <tt>-EINVAL</tt>?</p> <p>Hello,</p> <p>I was debugging the <a href="https://review.whamcloud.com/#/c/46955" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/46955</a> ("<a href="https://jira.whamcloud.com/browse/LU-15707" title="Unable to create file without a pool with a pool define on the parent or on the fs root." class="issue-link" data-issue-key="LU-15707"><del>LU-15707</del></a> lod: force creation of a component without a pool") when I saw the same message.</p> <p>I think the root cause is the following code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> <span class="code-keyword">static</span> <span class="code-keyword"><span class="code-object">int</span></span> lod_pool_add_q(<span class="code-keyword">struct</span> obd_device *obd, <span class="code-keyword"><span class="code-object">char</span></span> *poolname, <span class="code-keyword"><span class="code-object">char</span></span> *ostname) { <span class="code-keyword"><span class="code-object">int</span></span> err = lod_pool_add(obd, poolname, ostname); if (!err) { obd = obd_find_qmt0(obd-&gt;obd_name); &lt;------ if (obd) obd_pool_add(obd, poolname, ostname); } <span class="code-keyword">return</span> err; } </pre> </div></div> <p>If the mds0 has 2 MDT, the MGS will add 1 pool_add record in CONFIGS/lustre-MDT0000 and CONFIGS/lustre-MDT0001. So obd_pool_add/lod_pool_add_q will be call 2 times (1 for each MDT), QMT0000 will be found 2 times (on mds0) and finally qmt_pool_add/qmt_pool_add_rem will try to add the same OST on QMT0000.</p> <p>So maybe we can add/remove the pool to QMT0000 only for the MDT0000:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> <span class="code-keyword">static</span> <span class="code-keyword"><span class="code-object">int</span></span> lod_pool_add_q(<span class="code-keyword">struct</span> obd_device *obd, <span class="code-keyword"><span class="code-object">char</span></span> *poolname, <span class="code-keyword"><span class="code-object">char</span></span> *ostname) { <span class="code-keyword"><span class="code-object">int</span></span> err = lod_pool_add(obd, poolname, ostname); if (!err &amp;&amp; strstr(obd-&gt;obd_name, <span class="code-quote-red">"-MDT0000"</span>)) { obd = obd_find_qmt0(obd-&gt;obd_name); if (obd) obd_pool_add(obd, poolname, ostname); } <span class="code-keyword">return</span> err; } </pre> </div></div> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47059" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47059</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15055" title="spurious qmt_pool_add_rem() message when adding OST to pool: -EEXIST " class="issue-link" data-issue-key="LU-15055"><del>LU-15055</del></a> lod: run qmt_pool_* only from the MDT0000 config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9b4ea520fa3f05da47b20758a81360d03d25e2db</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47059/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47059/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15055" title="spurious qmt_pool_add_rem() message when adding OST to pool: -EEXIST " class="issue-link" data-issue-key="LU-15055"><del>LU-15055</del></a> lod: run qmt_pool_* only from the MDT0000 config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0f158c6a093e059d89f637f31d34742078c38209</p> <p>Landed for 2.16</p> Related LU-15043 Development Rank 1|i0263r: Rank (Obsolete) 9223372036854775807 Severity