https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15043 Lustre <p>Using the latest build of Lustre, 2.14.54_92 build # 4421, I created a spill pool loop and encountered some unexpected behavior. <br/> I created three pools and created a loop of spill pools where pool1.spill= pool2, pool2.spill=pool3 and pool3.spill=pool1. I then created a file on pool1, but the file was created on pool2. The same thing happened when I created a file on pool2 and on pool3, they were created on pool3 and pool1, respectively. </p> <p>I think we should not allow spill pool loops to be created. </p> <p>Here are more details:<br/> Created three pools: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs pool_list scratch.pool1 Pool: scratch.pool1 scratch-OST0000_UUID # lfs pool_list scratch.pool2 Pool: scratch.pool2 scratch-OST0001_UUID # lfs pool_list scratch.pool3 Pool: scratch.pool3 scratch-OST0002_UUID </pre> </div></div> <p>Set spill pool and thresholds on both MDSs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mds1# lctl get_param lod.scratch-MDT*.pool.*.spill* lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_target=pool3 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_threshold_pct=5 lod.scratch-MDT0000-mdtlov.pool.pool3.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool3.spill_target=pool1 lod.scratch-MDT0000-mdtlov.pool.pool3.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_target=pool3 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool3.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool3.spill_target=pool1 lod.scratch-MDT0002-mdtlov.pool.pool3.spill_threshold_pct=5 </pre> </div></div> <p>We see the following in dmesg on mds1, not on mds2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 9046.643396] LustreError: 5659:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0000_UUID pool pool1: rc = -17 [ 9056.957864] LustreError: 5666:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0001_UUID pool pool2: rc = -17 [ 9065.980468] LustreError: 5674:0:(qmt_pool.c:1390:qmt_pool_add_rem()) add to: can't scratch-QMT0000 scratch-OST0002_UUID pool pool3: rc = -17 </pre> </div></div> <p>Create files on specific OST pools:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs setstripe -p pool1 -c -1 /lustre/scratch/file1 # lfs getstripe -p /lustre/scratch/file1 pool2 # lfs setstripe -p pool2 -c -1 /lustre/scratch/file2 # lfs getstripe -p /lustre/scratch/file2 pool3 # lfs setstripe -p pool3 -c -1 /lustre/scratch/file3 # lfs getstripe -p /lustre/scratch/file3 pool1 </pre> </div></div> <p>We see the following on MDS0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10198.677195] Lustre: 1506:0:(lod_pool.c:799:lod_check_and_spill_pool()) scratch-MDT0000-mdtlov: more than 10 levels of pool spill for 'pool1-&gt;pool2' [10223.616652] Lustre: 1506:0:(lod_pool.c:799:lod_check_and_spill_pool()) scratch-MDT0000-mdtlov: more than 10 levels of pool spill for 'pool2-&gt;pool3' [10234.693511] Lustre: 1538:0:(lod_pool.c:799:lod_check_and_spill_pool()) scratch-MDT0000-mdtlov: more than 10 levels of pool spill for 'pool3-&gt;pool1' </pre> </div></div> LU-15043

OST spill pools should not allow spill pool loops

Bug Major Resolved Fixed Alex Zhuravlev James Nunez Tue, 28 Sep 2021 17:45:46 +0000 Mon, 27 Jun 2022 15:44:37 +0000 Mon, 27 Jun 2022 15:44:37 +0000 Lustre 2.15.0 Lustre 2.16.0 0 4 <p>James, the "<tt>more than 10 levels of pool spill</tt>" logic in the code is to prevent an infinite loop in the kernel as it follows the circular linked list of spill targets. It looks like it stops at the 10th level of pool spilling (3 full loops plus 1), which explains the <tt>-p pool1</tt> creating a file in <tt>pool2</tt>.</p> <p>Detecting a loop at the time <tt>spill_target</tt> is set should be relatively simple to implement. In <tt>lod_spill_target_seq_write()</tt> it should first copy the specified target into a temporary buffer, rather than <tt>pool_spill_target</tt>, and follow the specified target pool until it hits a pool with no <tt>spill_target</tt> set (the normal case), or the <tt>pool_spill_target</tt> is the same as <tt>pool-&gt;pool_name</tt>.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45083" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45083</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15043" title="OST spill pools should not allow spill pool loops" class="issue-link" data-issue-key="LU-15043"><del>LU-15043</del></a> lod: check for spilling loops<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14d441c9bdb4873e4b0255658873ee963828548e</p> <p>The <tt>qmt_pool_add_rem()</tt> message started appearing on 2021-08-18, and there have been a few thousand hits per day. Patches landed on that day are listed below (no other patches landed after 2021-08-10 or before 2021-08-25):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git log --oneline --before 2021-08-22 --after 2021-08-16 d8204f903a (tag: v2_14_54, tag: 2.14.54) New tag 2.14.54 5220160648 LU-14093 lutf: fix build with gcc10 a205334da5 LU-14903 doc: update lfs-setdirstripe man page 1313cad7a1 LU-14899 ldiskfs: Add 5.4.136 mainline kernel support c44afcfb72 LU-12815 socklnd: set conns_per_peer based on link speed 6e30cd0844 LU-14871 kernel: kernel update RHEL7.9 [3.10.0-1160.36.2.el7] 14b8276e06 LU-14865 utils: llog_reader.c printf type mismatch aa5d081237 LU-9859 lnet: fold lprocfs_call_handler functionality into lnet_debugfs_* e423a0bd7a LU-14787 libcfs: Proved an abstraction for AS_EXITING 76c71a167b LU-14775 kernel: kernel update SLES12 SP5 [4.12.14-122.74.1] 67752f6db2 LU-14773 tests: skip check_network() on working node 024f9303bc LU-14668 lnet: Lock primary NID logic 684943e2d0 LU-14668 lnet: peer state to lock primary nid 16321de596 LU-14661 obdclass: Add peer/peer NI when processing llog ac201366ad LU-14661 lnet: Provide kernel API for adding peers 51350e9b73 LU-14531 osd: serialize access to object vs object destroy a5cbe7883d LU-12815 socklnd: allow dynamic setting of conns_per_peer d13d8158e8 LU-14093 mgc: rework mgc_apply_recover_logs() for gcc10 8dd4488a07 LU-6142 tests: remove iam_ut binary 301d76a711 LU-14876 out: don't connect to busy MDS-MDS export </pre> </div></div> <p>The graph shows occurrences by subtest, it looks like this happens in any subtest that is adding a pool:<br/> <span class="image-wrap" style=""><a id="40743_thumb" href="https://jira.whamcloud.com/secure/attachment/40743/40743_Screen+Shot+2021-09-29+at+14.03.14.png" title="Screen Shot 2021-09-29 at 14.03.14.png" file-preview-type="image" file-preview-id="40743" file-preview-title="Screen Shot 2021-09-29 at 14.03.14.png"><img src="https://jira.whamcloud.com/secure/thumbnail/40743/_thumb_40743.png" style="border: 0px solid black" role="presentation"/></a></span> </p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45083/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45083/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15043" title="OST spill pools should not allow spill pool loops" class="issue-link" data-issue-key="LU-15043"><del>LU-15043</del></a> lod: check for spilling loops<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c9c842d678e38345c890c1514e9b922fe496dba7</p> <p>Landed for 2.16</p> Related LU-15011 LU-15055 Development Rank 1|i025ov: Rank (Obsolete) 9223372036854775807 Severity