https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10465 Lustre <p>Patch <a href="https://review.whamcloud.com/25336" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25336</a> for <a href="https://jira.whamcloud.com/browse/LU-9090" title="increase default RPC size to 4MB" class="issue-link" data-issue-key="LU-9090"><del>LU-9090</del></a> has increased the default OST BRW size to 4MB. Patch for this ticket will increase the default stripe size from 1MB to 4MB so that widely-striped files can generate full RPCs without pinning so much memory on the client.</p> LU-10465

increase default stripe size to 4MB

Bug Critical Resolved Fixed Mikhail Pershin Jian Yu LTS12 Mon, 8 Jan 2018 01:50:26 +0000 Fri, 5 Jan 2024 22:19:53 +0000 Sat, 18 Nov 2023 21:55:28 +0000 Lustre 2.11.0 Lustre 2.16.0 0 9 <p>Here is the patch to increase default stripe size to 4MB: <a href="https://review.whamcloud.com/27151" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27151</a></p> <p>Hi Cliff,</p> <p>Could you please test the above patch according to the following suggestion from Andreas?</p> <blockquote><p>Before this patch can be landed, we need to understand what kind of performance impact will be seen. Please have Cliff run a test on our system to verify e.g. IOR is running as fast or faster than before.</p></blockquote> <p>Thank you.</p> <p>Saurabh is doing performance testing now, we'll get this into the schedule. </p> <p>I will add this into my schedule and get back with results.</p> <p>In light of <a href="https://jira.whamcloud.com/browse/LU-10463" title="Poor write performance periodically on repeated test runs" class="issue-link" data-issue-key="LU-10463"><del>LU-10463</del></a>, I wonder if the default stripe size should be a function of the default RPC size?  The MDS could pick this up from the OSTs at connect time.  It might get more complex if there is a mix of ldiskfs and ZFS OSTs in a single system (though that is unlikely).</p> <p>That said, I think it is less harmful if the default stripe size is <b>larger</b> than the RPC size, than if the stripe size is <b>smaller</b> than the RPC size.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31292" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31292</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> tests: interoperate with 4MB stripe size server<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 2ec6a165cd23cdaeaf47ff17f09903450e866412</p> <p>Ihara or Vitaly, do you have any performance test results from testing with the default <b><tt>stripe_size</tt></b> of 4MB (not the RPC size)? Do you already run with this in production? We're just looking to see if the default should be increased. It looks like a good improvement for ZFS, but our testing for ldiskfs is mixed, so it would be good to get some more feedback from more systems if possible.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27151/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27151/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lov: increase default stripe size to 4MB<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3f5abc6fa30e7c0256077ccf6a149d1809450465</p> <p>Landed for 2.11</p> <p>Due to problems related to <a href="https://jira.whamcloud.com/browse/LU-10786" title="sanity-flr test_45: Create /mnt/lustre/d45.sanity-flr/f45.sanity-flr failed" class="issue-link" data-issue-key="LU-10786"><del>LU-10786</del></a>, we need to undo the 4MB default stripe size change until we have a better method of handling DoM components. Otherwise, it means that DoM files will not be created easily with default settings.</p> <p>Rather than revert the whole patch, I would recommend to submit a new patch that is only changing the default stripe size, and leave the test fixes in place. That allows developers to specify different default stripe size without hitting unrelated failures, and simplifies testing in the future.</p> <p>Once the patch to change the default stripe size back to 1MB has landed this ticket should be moved to 2.12.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31589" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31589</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lov: decrease default stripe size to 1MB<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fec94eb84185e5c790d41766f4cd9bd348e49a00</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31589/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31589/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lov: decrease default stripe size to 1MB<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b231e3ae5cf7df6abf1fbd683589f0fcff2de03f</p> <p>We've backed off changing the default for 2.11</p> <p>I think with <a href="https://jira.whamcloud.com/browse/LU-10808" title="DoM: component end should align with dom_stripesize" class="issue-link" data-issue-key="LU-10808"><del>LU-10808</del></a> fixed that we could try reintroducing this change (essentially revert <a href="https://review.whamcloud.com/31589" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31589</a> again).</p> <p>Ihara, if you get a chance, could you please run a 4MB default stripe size for IOR FPP and SSF to see the performance impact.</p> <p>Vitaly, it would also be good to know if this change improves or hurts performance on your systems before it becomes the default.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37318" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37318</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lov: decrease default stripe size to 1MB"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 973a7e7042eda757080884ead71468d575516402</p> <p>Mike, it looks like there are still failures in the DoM tests when the default stripe size is changed to 4MB. Could you please take a look.<br/> <a href="https://testing.whamcloud.com/test_sets/c361e79a-3e3d-11ea-bc97-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c361e79a-3e3d-11ea-bc97-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ===================== 23:32:52 (1579822372) lfs migrate: cannot create composite file '/mnt/lustre/d272c.sanity/.:VOLATILE:0000:5EB80E61': Invalid argument error: lfs migrate: /mnt/lustre/d272c.sanity/f272c.sanity: cannot create volatile file: Operation not permitted sanity test_272c: @@@@@@ FAIL: failed to migrate to the new composite layout </pre> </div></div> <p>Andreas, this happens because new component stripes are not aligned with an old ones. We have original file with 1MB DOM component and not defined second component, e.g. it become 4MB stripes by default. Test tries to migrate to PFL layout with 2MB as first component stripe and second component by default. With 1MB default it works, but with 4MB boundaries are not aligned. So test should be changed to migrate to the same first stripe size it seems.</p> <p>Meanwhile I wonder, is that OK that components are not aligned by stripe size in a file? E.g. for original file we have [0, 1MB) for DOM and then 4MB stripes, so whole file has stripes as 1MB, 4MB, 4, ... and each 4MB stripe is not aligned at 4MB from the file beginning. I am not sure is that a problem or not, but wouldn't be better to adjust new component stripe sizes to be aligned with that component start?</p> <p>The second chunk (start of second component, after DoM component) should just be a bit shorter, starting at 1MB and ending at 4MB, with a "hole" at the start where the DoM opponent is. Then the rest of the chunks in the second component would be properly sized/aligned at 4MB. </p> <p>It is done this way so that eg. the DoM data could be written into the second component without having to move all the later data. </p> <p>Test failed due to that error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lod_lov.c:1934:lod_verify_striping()) stripe size isn't aligned, stripe_sz: 4194304, [0, 2097152) </pre> </div></div> <p>So this check in <tt>lod_verify_striping()</tt> is not quite correct, isn't it? If original file has 1MB for first component and 3MB, 4MB ... for the second, then it can be migrated to 2MB for the first and then 2MB, 4MB, ... for the second component.<br/>  </p> <p>Probably I see where problem is, test does the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $LFS migrate -E 2M -c1 -E -1 -c2 $dom </pre> </div></div> <p>without implicitly set stripe size, <tt>LOD</tt> uses default size 4MB and fails because it is over whole component 2MB. Considering the user may not know default MDT stripe size, I'd say it is not his fault to use 2MB component, and either <tt>lfs</tt> or <tt>LOD</tt> should take care and reduce stripe size to the component size maybe?</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37661" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37661</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lod: adjust component stripe size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6b589349595efb4eba73b40c7e7e8f07c41bd8b3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37661/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37661/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lod: adjust component stripe size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e67a7f7b960a042ae7369e81e4365c6f7e095d25</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39957" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39957</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lod: adjust component stripe size<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: bf76de5430fa104b2992f40768cdf8c97699f1de</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/37318/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/37318/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> lov: increase default stripe size to 4MB<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ea18d7da59d369f093e340e150544f51b2f229a1</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52989" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52989</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> osd-ldiskfs: 8MiB IOs should bypass cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 22f24efa69e46eaad909403f3bb473b5d40cab32</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52989/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52989/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10465" title="increase default stripe size to 4MB" class="issue-link" data-issue-key="LU-10465"><del>LU-10465</del></a> osd-ldiskfs: 8MiB IOs should bypass cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f8e49e321ed81d77b204af40165f9ae2d07c5986</p> <p>Landed for 2.16</p> Duplicate Related LU-10463 LU-9090 LU-10786 LU-17076 LU-10808 LU-11918 Development Rank 1|hzzqm7: Rank (Obsolete) 9223372036854775807 Severity