https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16587 Lustre <p>Most of the time <tt>lfs migrate</tt> uses a 1MB buffer (stripe size) to copy data. This is terribly slow. </p> <p>In my testing I see improved performance up to 64M buffers.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@kjlmo4n00 16G]# lfs getstripe 16G.1 16G.1 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 11 lmm_stripe_offset: 1 lmm_pool: flash obdidx objid objid group 1 394422 0x604b6 0 [root@kjlmo4n00 16G]# time lfs migrate -S 1M -p flash 16G.1 real 0m25.341s user 0m0.001s sys 0m2.606s [root@kjlmo4n00 16G]# time /root/tools/lfs_nzr migrate -S 1M -p flash 16G.1 real 0m6.526s user 0m0.000s sys 0m2.177s </pre> </div></div> <p>You can force lfs to use a bigger buffer by increasing stripe size and you can see similar improvements.</p> LU-16587

Make lfs migrate faster

Improvement Minor Resolved Fixed Nathan Rutman Nathan Rutman Wed, 22 Feb 2023 22:32:21 +0000 Wed, 2 Aug 2023 18:47:42 +0000 Mon, 3 Apr 2023 21:45:09 +0000 0 7 <p>Sigh. Permission denied trying to upload to Gerritt; stashing patch here for now.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/utils/lfs.c b/lustre/utils/lfs.c index 999e6357a1..c2aa48d071 100644 --- a/lustre/utils/lfs.c +++ b/lustre/utils/lfs.c @@ -832,7 +832,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> migrate_copy_data(<span class="code-object">int</span> fd_src, <span class="code-object">int</span> fd_dst, <span class="code-object">int</span> (*check_file)(<span class="code-object">int</span>), <span class="code-object">long</span> stats_interval_sec, off_t file_size_bytes) { struct llapi_layout *layout; - size_t buf_size = 4 * 1024 * 1024; + size_t buf_size = 64 * 1024 * 1024; void *buf = NULL; off_t pos = 0; off_t data_end = 0; @@ -850,8 +850,14 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> migrate_copy_data(<span class="code-object">int</span> fd_src, <span class="code-object">int</span> fd_dst, <span class="code-object">int</span> (*check_file)(<span class="code-object">int</span>), uint64_t stripe_size; rc = llapi_layout_stripe_size_get(layout, &amp;stripe_size); - <span class="code-keyword">if</span> (rc == 0) - buf_size = stripe_size; + <span class="code-keyword">if</span> (rc == 0) { + <span class="code-comment">/* We like big bufs */</span> + <span class="code-keyword">if</span> (stripe_size &gt; buf_size) + buf_size = stripe_size; + <span class="code-keyword">else</span> + <span class="code-comment">/* Trim to stripe_size multiple */</span> + buf_size -= buf_size % stripe_size; + } llapi_layout_free(layout); } </pre> </div></div> <p>Nathan,</p> <p>Share your Gerrit error message here and we might be able to help?</p> <p>I sort of thought we'd already done this, when we switched it to using direct I/O...  Oops.  DIO won't help at all if the buffer size is still small...</p> <p>"Nathan Rutman &lt;nrutman@gmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50118" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50118</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16587" title="Make lfs migrate faster" class="issue-link" data-issue-key="LU-16587"><del>LU-16587</del></a> utils: give lfs migrate a larger buffer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f93e048409c6ea03dc2afe48788cf0a1fd0bbbed</p> <p>Patrick, do we default to O_DIRECT now that your optimizations are landed? Presumably that would only get faster over time, and be avoid thrashing the client page cache when migrating a lot of files. </p> <p>Well, now you've made me check...</p> <p>Yes - We default to direct I/O and let you turn it off with 'D' for non-direct mode.  And now that Nathan is increasing the buffer size, it will actually be faster.  Oops.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50118/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50118/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16587" title="Make lfs migrate faster" class="issue-link" data-issue-key="LU-16587"><del>LU-16587</del></a> utils: give lfs migrate a larger buffer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 23224e03dc30c89dd449de5a7fe99b0bd3aca495</p> <p>Landed for 2.16</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51115" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51115</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16587" title="Make lfs migrate faster" class="issue-link" data-issue-key="LU-16587"><del>LU-16587</del></a> utils: give lfs migrate a larger buffer<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 10a6cdb9359e4728b2541e70c07043ce10176f84</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51116" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51116</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16587" title="Make lfs migrate faster" class="issue-link" data-issue-key="LU-16587"><del>LU-16587</del></a> utils: give lfs migrate a larger buffer<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7c8f2aee8e5c87b5415c6ac20497ee5687372835</p> Related LU-16948 Development Rank 1|i03etb: Rank (Obsolete) 9223372036854775807