[LU-16587] Make lfs migrate faster

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Assignee: Nathan Rutman
• Reporter: Nathan Rutman

Description

Most of the time lfs migrate uses a 1MB buffer (stripe size) to copy data. This is terribly slow.

In my testing I see improved performance up to 64M buffers.

[root@kjlmo4n00 16G]# lfs getstripe 16G.1
16G.1
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    11
lmm_stripe_offset: 1
lmm_pool:          flash
	obdidx		 objid		 objid		 group
	     1	        394422	      0x604b6	             0

[root@kjlmo4n00 16G]# time lfs migrate -S 1M -p flash 16G.1

real	0m25.341s
user	0m0.001s
sys	0m2.606s
[root@kjlmo4n00 16G]# time /root/tools/lfs_nzr migrate -S 1M -p flash 16G.1

real	0m6.526s
user	0m0.000s
sys	0m2.177s

You can force lfs to use a bigger buffer by increasing stripe size and you can see similar improvements.

Issue Links

• is related to: LU-16948 sparse file migration broken
• is related to: LU-16789 "lfs migrate" to use AIO/DIO or io_uring (kernel 5.1+)

Comments

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51116
Subject: LU-16587 utils: give lfs migrate a larger buffer
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 7c8f2aee8e5c87b5415c6ac20497ee5687372835

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51115
Subject: LU-16587 utils: give lfs migrate a larger buffer
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 10a6cdb9359e4728b2541e70c07043ce10176f84

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50118/
Subject: LU-16587 utils: give lfs migrate a larger buffer
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 23224e03dc30c89dd449de5a7fe99b0bd3aca495

[Patrick Farrell]

Well, now you've made me check...

Yes - We default to direct I/O and let you turn it off with 'D' for non-direct mode.  And now that Nathan is increasing the buffer size, it will actually be faster.  Oops.

[Andreas Dilger]

Patrick, do we default to O_DIRECT now that your optimizations are landed? Presumably that would only get faster over time, and be avoid thrashing the client page cache when migrating a lot of files.

[Gerrit Updater]

"Nathan Rutman <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50118
Subject: LU-16587 utils: give lfs migrate a larger buffer
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f93e048409c6ea03dc2afe48788cf0a1fd0bbbed

[Patrick Farrell]

Nathan,

Share your Gerrit error message here and we might be able to help?

I sort of thought we'd already done this, when we switched it to using direct I/O...  Oops.  DIO won't help at all if the buffer size is still small...

[Nathan Rutman]

Sigh. Permission denied trying to upload to Gerritt; stashing patch here for now.

diff --git a/lustre/utils/lfs.c b/lustre/utils/lfs.c
index 999e6357a1..c2aa48d071 100644
--- a/lustre/utils/lfs.c
+++ b/lustre/utils/lfs.c
@@ -832,7 +832,7 @@ static int migrate_copy_data(int fd_src, int fd_dst, int (*check_file)(int),
                             long stats_interval_sec, off_t file_size_bytes)
 {
        struct llapi_layout *layout;
-       size_t buf_size = 4 * 1024 * 1024;
+       size_t buf_size = 64 * 1024 * 1024;
        void *buf = NULL;
        off_t pos = 0;
        off_t data_end = 0;
@@ -850,8 +850,14 @@ static int migrate_copy_data(int fd_src, int fd_dst, int (*check_file)(int),
                uint64_t stripe_size;
 
                rc = llapi_layout_stripe_size_get(layout, &stripe_size);
-               if (rc == 0)
-                       buf_size = stripe_size;
+               if (rc == 0) {
+                       /* We like big bufs */
+                       if (stripe_size > buf_size)
+                               buf_size = stripe_size;
+                       else
+                               /* Trim to stripe_size multiple */
+                               buf_size -= buf_size % stripe_size;
+               }
 
                llapi_layout_free(layout);
        }