https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13076 Lustre <p>When migrating directory trees to new MDTs for space balancing, it would be very convenient to allow specifying "<tt>lfs migrate -m -1</tt>" to have "<tt>lfs</tt>" pick the MDT with the most free inodes as the target. This is similar to "<tt>lfs setdirstripe -i -1</tt>" functionality, but for migration. If multiple directories are specified on the command line, it probably makes sense to refresh the statfs information after each directory tree is migrated, in case there is a new MDT that has more free space. This will greatly simplify directory migration on an existing filesystem without the user having to specify the details.</p> <p>If this is already implemented (it didn't look like <tt>parse_targets()</tt> would handle <tt><del>1</tt> as an argument), then the <tt>lfs-migrate.1</tt> man page needs to be updated with this information under the description of <tt></del>-mdt-index</tt>.</p> <p>I suspect this would be relatively easily implemented in Lustre 2.12, because the mechanism to select the best MDT is already present in <tt>lfs</tt>.</p> LU-13076

DNE3: lfs migrate -m should allow -1 as the target index

Improvement Minor Resolved Fixed Lai Siyao Andreas Dilger dne3 Fri, 13 Dec 2019 13:11:10 +0000 Sat, 4 Dec 2021 06:46:53 +0000 Wed, 27 Oct 2021 03:57:00 +0000 Lustre 2.15.0 0 5 <p>'lfs migrate -m -1' is not allowed in 2.12, and in the implementation of <a href="https://jira.whamcloud.com/browse/LU-11025" title="DNE3: directory restripe" class="issue-link" data-issue-key="LU-11025"><del>LU-11025</del></a>, 'lfs migrate -m -1' will do directory split/merge instead of migration, though for directory split it will choose the MDTs that have more free space.</p> <p>Doesn't it make more sense to have "<tt>lfs migrate -m N -c -1</tt>" do directly split, or similar? The "<tt>-m</tt> is used as the DNE stripe index, not the stripe count. </p> <p>"lfs migrate -m N -c -1" doesn't look meaningful to me: migrate directory to MDT N and with stripe count -1? Maybe we can add a new command "lfs restripe -c -H &lt;dir&gt;" for directory split/merge.</p> <p>BTW let system choose target MDTs may not be optimal in some cases:</p> <ul> <li>case 1: dir1 is fairly big, and it's a plain directory on MDT0, now system decides to migrate it to MDT1 and MDT2, though currently they have more free space than MDT0, after migration, they may become more full, or then don't have enough space to finish migration.</li> <li>case 2: dir1 is located on MDT0, and system decides to migrate to MDT1 and MDT2, this will cause all sub files/dirs moved, in contrast, if user manually migrate it to MDT0 and MDT2, only half sub files/dirs needs to be moved.</li> </ul> <p>Ah, I think I understand where my confusion is. Am I correct that you are separating the use of "<tt>lfs migrate -m N</tt>" to mean "migrate parent directory with all inodes to new MDT 'N'" and "<tt>lfs migrate -m N -c -1</tt>" to mean the new "split parent directory but leave inodes in place" code that you are working on? I think that "migrate" should generally mean "move inodes to new MDT", while "MDT auto split" should not move existing inodes since that will change inode numbers/locks and could cause problems. I think it is useful to have the meaning of "<tt>lfs migrate -m N -c C</tt>" for directories be similar to "<tt>lfs migrate -i N -c C</tt>" for regular files, where "<tt>N = -1</tt>" means "pick the target index for me" and "<tt>C = -1</tt>" means "stripe over all targets".</p> <p>Being able to specify "split directory now" is important for testing, but maybe a different command like "<tt>lfs setstripe -c &lt;dir&gt;</tt>" on the existing directory is the right command for splitting the directory to have a different number of stripes? For <tt>stripe_count=1</tt> directories it is possible to add any kind of hash function to the existing directory, so "<tt>crush</tt>" would be preferred, and then setting the number of stripes would just migrate the names without affecting the inodes?</p> <p>I also agree that a simple implementation of "<tt>lfs migrate -m -1</tt>" might not pick the best MDTs, but users (other than you and me) are even <em>less</em> likely to pick the best MDTs. That means that the implementation of "<tt>-m -1</tt>" needs to be smart enough that it picks good MDTs when possible.</p> <p>It should be possible to disable MDTs for new directory creation (similar to "<tt>lfs set_param osp.$fsname-OST0000.max_create_count=0</tt>" for OSTs) so that the MDS does not put new directories on that MDT. That will allow an MDT to be emptied out with a command like "<tt>lfs find -type d -m N | lfs migrate -m -1</tt>" without the user having to specify the target for every directory. </p> <p>Also, when the user is doing MDT balancing, the use of "<tt>lfs migrate -m -1</tt>" needs to move <b>some</b> directories and inodes to the empty MDTs instead of just splitting existing directories onto new MDTs and not moving any inodes, otherwise the full MDT will not be any less full. Maybe this can be done by weighting the current MDT higher than other MDTs but not keep all directories/inodes on the original MDT (e.g. using <tt>qos_mdt_prio_free</tt>)?</p> <p>The problem of using "lfs setstripe -c &lt;dir&gt;" (or should be "lfs setdirstripe -c &lt;dir&gt;"?) is this command is currently used to create new striped directories, if it needs to support dir split, it needs to verify whether target directory exists, which will downgrade striped directory creation performance. Do you think it's acceptable?</p> <p>Besides, to achieve best performance after migration, IMO it should be done in user space by policy engine, which is more flexible and easier to customize.</p> <p>You are correct that I meant "<tt>setdirstripe -c</tt>". It is fine if it "checks" whether the directory exists by trying to create it, then if this fails with <tt>-EEXIST</tt> or <tt>-EISDIR</tt> it can try to change the stripe count with a second RPC. I don't think this would impact the performance of "<tt>setdirstripe -c</tt>", or at least not in any critical way because there will likely be many more RPCs to migrate the entries. </p> <p>I also agree that a user space policy engine might be able to do a more optimal job than the MDS, but it should at least be possible to a basic job of directory migration without a policy engine. </p> <p>We currently have a customer with 24 MDTs, they used -c 24 for all their directories and now wish to re-stripe with -c 1. Given the customer lacks experience, I don't want them to have to manually choose a new target with lfs migrate -m - it would be very good to have -m -1, we need to have a hands-free restriping here. </p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44886" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44886</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13076" title="DNE3: lfs migrate -m should allow -1 as the target index" class="issue-link" data-issue-key="LU-13076"><del>LU-13076</del></a> dne: dir migrate in QOS mode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f4ee9c0f45d5031942f1238e7af212a4cee9f912</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44886/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44886/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13076" title="DNE3: lfs migrate -m should allow -1 as the target index" class="issue-link" data-issue-key="LU-13076"><del>LU-13076</del></a> dne: dir migrate in QOS mode<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 378c7567876b430d06031f7d380112b9bdb15166</p> <p>Landed for 2.15</p> Related LU-14210 LU-14975 Development Rank 1|i00r07: Rank (Obsolete) 9223372036854775807