https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15072 Lustre <p>I have an empty pool that consists on one OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs df -p scratch.pool1 UUID 1K-blocks Used Available Use% Mounted on scratch-MDT0000_UUID 17423180 3536 15848428 1% /lustre/scratch[MDT:0] scratch-MDT0001_UUID 17423180 2860 15849104 1% /lustre/scratch[MDT:1] scratch-MDT0002_UUID 27541652 3680 25055796 1% /lustre/scratch[MDT:2] scratch-MDT0003_UUID 27541652 3692 25055784 1% /lustre/scratch[MDT:3] scratch-OST0002_UUID 35055368 1672 33202304 1% /lustre/scratch[OST:2] filesystem_summary: 35055368 1672 33202304 1% /lustre/scratch </pre> </div></div> <p>On the MDSs, I set spill_threshold_pct to 6% and spilling is not active:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param lod.scratch-MDT*-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=6 # lctl get_param lod.scratch-MDT*.pool.pool1.* lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=6 </pre> </div></div> <p>I then change the spill_theshold_pct to 5% and spilling is activated. Note that no data was written to pool1 nor to the OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param lod.scratch-MDT*-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=5 # lctl get_param lod.scratch-MDT*.pool.pool1.* lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=5 </pre> </div></div> <p>Change the threshold back to 6% and spilling is not active again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param lod.scratch-MDT*-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=6 # lctl get_param lod.scratch-MDT*.pool.pool1.* lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=6 </pre> </div></div> <p>With an empty OST pool, pool spilling should not be activated when the threshold is changed and the values are all above 1%.</p> LU-15072

Pool spill is activated just by changing threshold

Bug Critical Resolved Fixed Alex Zhuravlev James Nunez Thu, 7 Oct 2021 15:02:53 +0000 Sat, 22 Apr 2023 18:33:07 +0000 Sat, 22 Apr 2023 18:33:07 +0000 Lustre 2.15.0 Lustre 2.16.0 0 7 <p>I'm adding another status to procfs to see statfs data spilling code sees. that may be related to grants.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45153" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45153</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15072" title="Pool spill is activated just by changing threshold" class="issue-link" data-issue-key="LU-15072"><del>LU-15072</del></a> lod: extra status for spilling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0fa9a39022741bc7f523a97d39328688c00023b2</p> <p>James, thanks for the report. could you please apply the patch above and dump spill internal info via </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl get_param lod.*.pool.*.spill_status</pre> </div></div> <p> ?</p> <p>Just created pools pool0, pool1, pool2 with assignment OST0000, OST0001 and OST00002, respectively. No data is written to any pool and nothing written to the file system:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param lod.scratch-MDT*.pool.*.spill* lod.scratch-MDT0000-mdtlov.pool.pool0.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_status= 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_target= lod.scratch-MDT0000-mdtlov.pool.pool0.spill_threshold_pct=0 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_status= 1: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target= lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=0 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_status= 2: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0000-mdtlov.pool.pool2.spill_threshold_pct=0 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_status= 0: 0 0 0 - 0 = 0 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_target= lod.scratch-MDT0002-mdtlov.pool.pool0.spill_threshold_pct=0 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_status= 1: 0 0 0 - 0 = 0 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target= lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=0 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_status= 2: 0 0 0 - 0 = 0 ?? 0 -- off 0% expired -14015 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0002-mdtlov.pool.pool2.spill_threshold_pct=0 </pre> </div></div> <p>Set poolN.spill_target=pool(N+1) for N = 0, 1 and set spill_threshold_pct=5:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param lod.scratch-MDT*.pool.*.spill* lod.scratch-MDT0000-mdtlov.pool.pool0.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_status= 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_target=pool1 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_threshold_pct=5 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_status= 1: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_is_active=1 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_status= 2: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0000-mdtlov.pool.pool2.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_status= 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_target=pool1 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_status= 1: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=5 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_is_active=1 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_status= 2: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 939308 -- on 5% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0002-mdtlov.pool.pool2.spill_threshold_pct=5 </pre> </div></div> <p>Set spill_threshold_pct=6:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl get_param lod.scratch-MDT*.pool.*.spill* lod.scratch-MDT0000-mdtlov.pool.pool0.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_status= 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_target=pool1 lod.scratch-MDT0000-mdtlov.pool.pool0.spill_threshold_pct=6 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_status= 1: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0000-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_is_active=0 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_status= 2: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0000-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0000-mdtlov.pool.pool2.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_status= 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_target=pool1 lod.scratch-MDT0002-mdtlov.pool.pool0.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_status= 1: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_target=pool2 lod.scratch-MDT0002-mdtlov.pool.pool1.spill_threshold_pct=6 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_is_active=0 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_status= 2: 17771692 18786160 18786160 - 17771692 = 1014468 ?? 1127169 -- off 6% expired 5 lod.scratch-MDT0002-mdtlov.pool.pool2.spill_target= lod.scratch-MDT0002-mdtlov.pool.pool2.spill_threshold_pct=6 </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 0: 17771696 18786160 18786160 - 17771696 = 1014464 ?? 939308 -- on 5% expired 5 </pre> </div></div> <p>total is 18345 MB and 17355MB is available, thus used is 990MB or 5.4%<br/> just mechanically it's all correct, but I guess this can be confusing for customers.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 122.1M 1.6M 109.5M 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 305.8M 1.2M 278.0M 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 305.8M 1.2M 278.0M 1% /mnt/lustre[OST:1] </pre> </div></div> <p>this is on a local tiny setup where we do not create a huge journal.</p> <p>important notice - status is not recalculated just when space usage changes, rather it's done upon object allocation (e.g. when you create a new file/mirror) and changing spilling configuration.</p> <p>Alex, I think it makes sense to compute the used percentage in the same way as "<tt>df</tt>" and "<tt>lfs df</tt>" does, so that users are not confused by this. </p> <p>OK, I'll check the details </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jamesanunez" class="user-hover" rel="jamesanunez">jamesanunez</a> could you please show lfs df for the case?</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45153/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45153/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15072" title="Pool spill is activated just by changing threshold" class="issue-link" data-issue-key="LU-15072"><del>LU-15072</del></a> lod: extra status for spilling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b7f07db28f6da76b56f53988866f0cc6cda158d3</p> <p>Landed for 2.16</p> Related LU-15011 Development Rank 1|i026lz: Rank (Obsolete) 9223372036854775807 Severity