https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12335 Lustre <p>Preallocation table read/write code is racy. There is a  possibility of accessing memory outside of allocated table.<br/> This issue can be easy reproduced. I am not sure, I have to upload test that lead to test system to be crashed. So I put it here.<br/>  <br/> dd if=/dev/zero of=&lt;path_to_ldiskfs_partition&gt; bs=1048576 count=1024 conv=fsync<br/> cat "32 64 128 256" &gt; /proc/fs/ldiskfs/&lt;dev&gt;/prealloc_table</p> LU-12335

mb_prealloc_table table read/write code is racy

Bug Critical Resolved Fixed Artem Blagodarenko Artem Blagodarenko Fri, 24 May 2019 07:25:18 +0000 Tue, 10 Sep 2019 14:00:42 +0000 Tue, 10 Sep 2019 13:45:39 +0000 Lustre 2.13.0 0 3 <p>Artem Blagodarenko (c17828@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/34950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12335" title="mb_prealloc_table table read/write code is racy" class="issue-link" data-issue-key="LU-12335"><del>LU-12335</del></a> ldiskfs: fixed size preallocation table<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 411f62650ddb2ef569fbcd38891e0610ce44b0d3</p> <p>I guess the first question is whether the preallocation table settings are even useful? We've been carrying that patch for many years without submitting it upstream, because I'm not sure whether it actually improves performance or functionality or is just overhead for patch maintenance? Do you have a real test system where you could measure performance under load to see if removing <tt>ext4-prealloc.patch</tt> improves or hurts performance or allocation behaviour? </p> <p>If there is data that shows the patch improves performance noticeably under at least some non-Lustre workloads, and doesn't hurt performance, then it would make sense to push the patch upstream finally.</p> <p>I used preallocation table to solve allocator problems on aged systems(<a href="https://jira.whamcloud.com/browse/LU-12103" title="Improve block allocation for large partitions" class="issue-link" data-issue-key="LU-12103"><del>LU-12103</del></a>). There are two (and 3rd is bigalloc) solutions: </p> <ul> <li>A new block allocator algorithm has been developed (<a href="https://jira.whamcloud.com/browse/LU-12103" title="Improve block allocation for large partitions" class="issue-link" data-issue-key="LU-12103"><del>LU-12103</del></a>, send to upstream) by Cray to strategically skip low probability-of-match block groups while attempting to locate contiguous block groups when they likely won’t exist.  </li> <li>A novel script has been developed to dynamically adjust the block device pre-allocation table. This controls the number of pre-allocated blocks that are created for the request size in logarithmic increments starting at 4. As file systems fragment and become filled, some free block groups will simply not be available. Because of this, the block allocator should be tuned to address this on a regular basis.  </li> </ul> <blockquote><p>Do you have a real test system where you could measure performance under load to see if removing <tt>ext4-prealloc.patch</tt> improves or hurts performance or allocation behaviour?</p></blockquote> <p>We have test results for third solution. 140TB ldiskfs partition. Will share results to <a href="https://jira.whamcloud.com/browse/LU-12103" title="Improve block allocation for large partitions" class="issue-link" data-issue-key="LU-12103"><del>LU-12103</del></a>. For second solution I have some synthetic test results:</p> <p>Here is bash script that build prealloc table based on mb_groups output: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@localhost cray-lustre]# cat build_prealloc.sh  #!/bin/bash  INPUT_FILE=$1  #columes from 9 to 21 shows how free fragments available  <span class="code-keyword">for</span> index in {9..21}  <span class="code-keyword">do</span>     PARAMS=<span class="code-quote">"<span class="code-quote">'NR&gt;1 {<span class="code-keyword">if</span> (\$$index &gt; 0) { print }}'</span>"</span>  REGS=`eval awk <span class="code-quote">"$PARAMS"</span> $INPUT_FILE | wc -l`     VAL=$((2 ** ($index-8)))       [ $REGS -gt 0 ] &amp;&amp; PREALLOC_TABLE=<span class="code-quote">"$PREALLOC_TABLE $VAL"</span>  done  echo <span class="code-quote">"prealloc table: $PREALLOC_TABLE"</span>  </pre> </div></div> <p>Example how to use it: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cat /proc/fs/ldiskfs/loop1/mb_groups &gt; table.dat  sh build_prealloc.sh table.dat &gt; prealloc.txt  cat prealloc.txt &gt; /proc/fs/ldiskfs/loop1/prealloc_table  </pre> </div></div> <p> <br/> Here is test output of my local testing on shared fsxfs-n24.img. I have extracted and make two copies of this image for clear testing. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar -xf fsxfs-n24.img.tgz  cp fsxfs-n24.img fsxfs-n24-2.img  </pre> </div></div> <p>And run test that 1) make large preallocation table 2) start dd 3) adjust preallocation table using script above 4) start dd </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> start_mb_stats()  {          echo <span class="code-quote">"1"</span> &gt; /sys/fs/ldiskfs/loop1/mb_stats          echo <span class="code-quote">"0"</span> &gt; /sys/fs/ldiskfs/loop1/mb_c1_threshold          echo <span class="code-quote">"0"</span> &gt; /sys/fs/ldiskfs/loop1/mb_c2_threshold          echo <span class="code-quote">"0"</span> &gt; /sys/fs/ldiskfs/loop1/mb_c3_threshold  }  mount_image()  {          local IMAGE=$1          mount -t xfs -o loop $IMAGE /mnt/fs2xfs/          mount -t ldiskfs -o loop /mnt/fs2xfs/n24.raw /mnt/fs2ost/  }  umount_image()  {          umount /mnt/fs2ost/          umount /mnt/fs2xfs/  }  </pre> </div></div> <p>1. Set too large preallocation table and estimate write speed </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LOAD=yes lustre/tests/llmount.sh  mount_image /lustre/mnt/staff/CAST-19722/fsxfs-n24.img  echo "256 512 1024 2048 4096 8192 16384" &gt; /proc/fs/ldiskfs/loop1/prealloc_table  start_mb_stats  dd if=/dev/zero of=/mnt/fs2ost/O/foofile bs=1048576  count=1024  conv=fsync  cat /proc/fs/ldiskfs/loop1/mb_alloc  echo "clear" &gt; /proc/fs/ldiskfs/loop1/mb_alloc  umount_image  mount_image /lustre/mnt/staff/CAST-19722/fsxfs-n24-2.img  </pre> </div></div> <p>2. Adjast preallocation table based on mb_groups output </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cat /proc/fs/ldiskfs/loop1/mb_groups &gt; $TMP/table.dat  sh build_prealloc.sh $TMP/table.dat &gt; $TMP/prealloc.txt  cat $TMP/prealloc.txt &gt; /proc/fs/ldiskfs/loop1/prealloc_table  </pre> </div></div> <p>3. Estimate preformance again </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd if=/dev/zero of=/mnt/fs2ost/O/foofile bs=1048576  count=1024  conv=fsync  cat /proc/fs/ldiskfs/loop1/mb_alloc  echo "clear" &gt; /proc/fs/ldiskfs/loop1/mb_alloc  umount_image  [root@localhost cray-lustre]# sh start.sh   Loading modules from /lustre/mnt/orig/cray-lustre/lustre/tests/..  detected 8 online CPUs by sysfs  libcfs will create CPU partition based on online CPUs  1024+0 records in  1024+0 records out  1073741824 bytes (1.1 GB) copied, 11.2427 s, 95.5 MB/s  mballoc: 262144 blocks 153 reqs (137 success)  mballoc: 2046 extents scanned, 127 goal hits, 1 2^N hits, 10 breaks, 0 lost  mballoc: (0, 0, 0) useless c(0,1,2) loops  mballoc: (0, 0, 0) skipped c(0,1,2) loops  1024+0 records in  1024+0 records out  1073741824 bytes (1.1 GB) copied, 9.22825 s, 116 MB/s  mballoc: 262143 blocks 243 reqs (240 success)  mballoc: 141 extents scanned, 113 goal hits, 129 2^N hits, 0 breaks, 0 lost  mballoc: (0, 0, 0) useless c(0,1,2) loops  mballoc: (0, 0, 0) skipped c(0,1,2) loops  [root@localhost cray-lustre]#  </pre> </div></div> <p>test passed and shows ~18% speed improvement </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@localhost cray-lustre]# sh start.sh  Loading modules from /lustre/mnt/orig/cray-lustre/lustre/tests/..  detected 8 online CPUs by sysfs  libcfs will create CPU partition based on online CPUs  1024+0 records in  1024+0 records out  1073741824 bytes (1.1 GB) copied, 11.2427 s, 95.5 MB/s  mballoc: 262144 blocks 153 reqs (137 success)  mballoc: 2046 extents scanned, 127 goal hits, 1 2^N hits, 10 breaks, 0 lost  mballoc: (0, 0, 0) useless c(0,1,2) loops  mballoc: (0, 0, 0) skipped c(0,1,2) loops  1024+0 records in  1024+0 records out  1073741824 bytes (1.1 GB) copied, 9.22825 s, 116 MB/s  mballoc: 262143 blocks 243 reqs (240 success)  mballoc: 141 extents scanned, 113 goal hits, 129 2^N hits, 0 breaks, 0 lost  mballoc: (0, 0, 0) useless c(0,1,2) loops  mballoc: (0, 0, 0) skipped c(0,1,2) loops  [root@localhost cray-lustre]#  </pre> </div></div> <p>I am going test this approach on 140TB ldiskfs OST soon.</p> <blockquote> <p>A novel script has been developed to dynamically adjust the block device pre-allocation table. This controls the number of pre-allocated blocks that are created for the request size in logarithmic increments starting at 4. As file systems fragment and become filled, some free block groups will simply not be available. Because of this, the block allocator should be tuned to address this on a regular basis. </p></blockquote> <p>How hard would it be to include this into the mballoc code in the kernel directly? Having a userspace tool is OK, but suffers from a number of limitations (hard for most users to configure, can die if there are problems (e.g. OOM), become CPU starved if the server is busy, needs extra scanning to learn current filesystem state and may become out of sync with the kernel).</p> <p>The algorithm is not difficult, as you can see in script. So, can be added to kernel. The most difficult diction - moment then we need to reconfigure preallocation table. With script, administrator decide, then change configuration.</p> <p> </p> <p>&gt; (hard for most users to configure, can die if there are problems (e.g. OOM),</p> <p>My suggestion, add to cluster scripts and adjust automatically.</p> <p>&gt;become CPU starved if the server is busy</p> <p>Preallocation table changing is quite fast operation, and with patch <a href="https://review.whamcloud.com/34950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34950</a>, safe and lockless.</p> <p>&gt;needs extra scanning to learn current filesystem state and may become out of sync with the kernel).</p> <p>Scanning is made by kernel. Script use "/proc/fs/ldiskfs/loop1/mb_groups" output. This statistic is perfect data for such decision. Anyway, even in kernel we need use this statistic.</p> <p> </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34950/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34950/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12335" title="mb_prealloc_table table read/write code is racy" class="issue-link" data-issue-key="LU-12335"><del>LU-12335</del></a> ldiskfs: fixed size preallocation table<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f15995b8e52bafabe55506ad2e12c8a64a373948</p> <p>Landed for 2.13</p> <p>This fix was only every applied to RHEL platforms. SLES and Ubuntu lack this fix.</p> <p>As per recent LWG discussion this ticket should be marked as RESOLVED and anyone wanting to keep SLES/Ubuntu servers in sync should do that under a separate ticket</p> <p>Please make sure this is push to the ext4 maintainers.</p> <p>No need send this patch to ext4 upstream because no such bug there. Bug was introduced in our ldiskfs patches.</p> Related LU-12103 LU-12511 Development Rank 1|i00gx3: Rank (Obsolete) 9223372036854775807 Severity