https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13492 Lustre <p>Hello!</p> <p>When using <tt>lfs migrate -m</tt> to migrate directories across MDTs, we sometimes face <a href="https://jira.whamcloud.com/browse/LU-13298" title="lfs migrate -m &quot;migrate failed: Operation not supported (-95)&quot; on DoM files" class="issue-link" data-issue-key="LU-13298"><del>LU-13298</del></a> (lfs migrate does not work yet with DoM files) for which we do have a workaround (ie. we restripe the files first without DoM). However, we are now having a different problem this time, I think.</p> <p>We're trying to migrate files from MDT0003 to MDT0001. While running a migration of a full user directory as follow:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs migrate -m 1 /fir/users/apatel6 </pre> </div></div> <p>we hit "operation not permitted" errors on multiple directories, and even retrying the migration is leading to the same error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# lfs migrate -m 1 /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N migrate failed: Operation not permitted (-1) [root@fir-rbh01 storage]# lfs getdirstripe /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 3 [0x2800394ad:0x3c7c:0x0] 3 [0x280038894:0x124ee:0x0] </pre> </div></div> <p>I also noticed when writing this ticket that something seems wrong here as there are two mdtidx = "3". Usually, when a directory is migrating from 3 to 1, we can see mdtidx 1 and 3.</p> <p>Quick check of the FIDs above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# lfs fid2path /fir 0x2800394ad:0x3c7c:0x0 /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N [root@fir-rbh01 storage]# lfs fid2path /fir 0x280038894:0x124ee:0x0 /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N </pre> </div></div> <p>MDT0001 (not MDT0003!) shows this log message when attemping the failed command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 29 08:35:06 fir-md1-s2 kernel: LustreError: 22437:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. </pre> </div></div> <p>I don't see anything else, but there might be debug flags that could be interesting?<br/> In any case, let me know how we could help troubleshoot this issue. We're using Lustre 2.12.4 here even on the client that performs the lfs migrate. Thanks!</p> CentOS 7.6 Kernel 3.10.0-957.27.2.el7_lustre.pl2.x86_64 LU-13492

lfs migrate -m returns Operation not permitted

Bug Major Open Unresolved Hongchao Zhang Stephane Thiell Wed, 29 Apr 2020 15:51:26 +0000 Wed, 4 Oct 2023 18:27:07 +0000 Lustre 2.12.4 0 5 <p>Hongchao</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Stephane, are you able to collect debug logs from the client and MDS during the failed migration? Ideally, full debug in the client and MDS, but if the MDS is busy this would overflow the debug log, so if needed we could start with "<tt>debug=+dlmtrace+rpctrace</tt>".</p> <p>Thanks! Attached full debug (+ALL) from the client as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34797/34797_client-ALL.log" title="client-ALL.log attached to LU-13492">client-ALL.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> (client NID is 10.0.10.3@o2ib7) while running the following command (same as in the description above):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs migrate -m 1 /fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N </pre> </div></div> <p>and 2 seconds of debug logs from the two MDS in question:</p> <ul> <li>MDT0001 in <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34798/34798_fir-md1-s2-MDT0001_dlmtrace%2Brpctrace.log.gz" title="fir-md1-s2-MDT0001_dlmtrace+rpctrace.log.gz attached to LU-13492">fir-md1-s2-MDT0001_dlmtrace+rpctrace.log.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> <li>MDT0003 in <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34799/34799_fir-md1-s4-MDT0003_dlmtrace%2Brpctrace.log.gz" title="fir-md1-s4-MDT0003_dlmtrace+rpctrace.log.gz attached to LU-13492">fir-md1-s4-MDT0003_dlmtrace+rpctrace.log.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> </ul> <p>In the logs of MDT0001, I can see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00020000:16.0:1588205266.457978:0:22469:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: '02-N' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 02-N' to finish migration. </pre> </div></div> <p>so I think I got this part at least.</p> <p>Let me know if I should try full debug of the MDS. Perhaps I could increase the debug buffer size.</p> <p>We also noticed another thing on another directory tree, that may be related to this ticket.</p> <p>We were not able to migrate some "leaf" directories, and we noticed that all of them are actually empty.</p> <p>But even an explicit lfs migrate on them doesn't work (tested from both 2.12.4 and 2.13 clients):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# lfs getdirstripe /fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03 lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none [root@fir-rbh01 storage]# lfs migrate -m 1 /fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03 [root@fir-rbh01 storage]# echo $? 0 [root@fir-rbh01 storage]# lfs getdirstripe /fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03 lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none </pre> </div></div> <p>This directory is empty:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# stat /fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03 File: ‘/fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: e64e03a8h/3863872424d Inode: 180148089774940567 Links: 2 Access: (2755/drwxr-sr-x) Uid: (55081/ jbboin) Gid: (24300/ bgirod) Access: 2020-04-30 13:28:31.000000000 -0700 Modify: 2019-11-29 22:10:47.000000000 -0800 Change: 2019-11-29 22:10:47.000000000 -0800 Birth: - [root@fir-rbh01 storage]# ls -lisa /fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03 total 1840 180148089774940567 4 drwxr-sr-x 2 jbboin bgirod 4096 Nov 29 22:10 . 180148089774940559 1836 drwxr-sr-x 13322 jbboin bgirod 1871872 Apr 30 11:29 .. [root@fir-rbh01 storage]# </pre> </div></div> <p>Originally, we ran <tt>lfs migrate -m 1 /fir/groups/bgirod</tt>, which is mostly done by now, apart from a few empty directories in <tt>/fir/groups/bgirod/action_recognition/frames/</tt>.</p> <p>Now, if I try again, I get:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# lfs migrate -m 1 /fir/groups/bgirod /fir/groups/bgirod/ migrate failed: Operation not permitted (-1) </pre> </div></div> <p>And same error, on MDT0001:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fir-md1-s2: Apr 30 13:46:47 fir-md1-s2 kernel: LustreError: 22427:0:(mdd_dir.c:4496:mdd_migrate()) fir-MDD0001: 'bgirod' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H 2 bgirod' to finish migration. </pre> </div></div> <p>current getdirstripe info of each component:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 storage]# lfs getdirstripe /fir/groups/bgirod lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 3 [0x28003bb05:0x135:0x0] 1 [0x2400576a9:0x1abb:0x0] [root@fir-rbh01 storage]# lfs getdirstripe /fir/groups/bgirod/action_recognition lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 3 [0x28003bb05:0x136:0x0] 1 [0x2400576a9:0x1af9:0x0] [root@fir-rbh01 storage]# lfs getdirstripe /fir/groups/bgirod/action_recognition/frames lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 3 [0x28003bb05:0x138:0x0] 1 [0x2400576a9:0x1ce2:0x0] </pre> </div></div> <p>As per the stripe information of "/fir/groups/bgirod", "/fir/users/apatel6/data/10-scalingNEB/01-relaxwater/02-N", etc<br/> It should be the migration to MDT0003 (if the original directory was on MDT0003, there will be two mdtidx 3 in the stripes)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zhanghc tests]# ../utils/lfs getdirstripe /mnt/lustre/pdir/cdir/ lmv_stripe_count: 2 lmv_stripe_offset: 3 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 3 [0x2c0000400:0xb:0x0] 3 [0x2c0000400:0x9:0x0] </pre> </div></div> <p>Was there ever migration to MDT0003 prior to this migration?</p> <p>the "-EPERM" is triggered in "mdd_migrate" because of the pending migration</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int mdd_migrate(const struct lu_env *env, struct md_object *md_pobj, struct md_object *md_sobj, const struct lu_name *lname, struct md_object *md_tobj, struct md_op_spec *spec, struct md_attr *ma) { if (S_ISDIR(attr-&gt;la_mode)) { ... if (lmv-&gt;lmv_migrate_offset != lum_stripe_count || lmv-&gt;lmv_master_mdt_index != lmu-&gt;lum_stripe_offset || (lmv_hash_type != 0 &amp;&amp; lmv_hash_type != lmu-&gt;lum_hash_type)) { CERROR("%s: \'"DNAME"\' migration was " "interrupted, run \'lfs migrate " "-m %d -c %d -H %d "DNAME"\' to " "finish migration.\n", mdd2obd_dev(mdd)-&gt;obd_name, PNAME(lname), le32_to_cpu( lmv-&gt;lmv_master_mdt_index), le32_to_cpu( lmv-&gt;lmv_migrate_offset), le32_to_cpu(lmv_hash_type), PNAME(lname)); GOTO(out, rc = -EPERM); } ... } ... } </pre> </div></div> <p>The migration request will be sent to the migration target MDT, then the above log was printed at MDT0001</p> <p>For the empty directory issue of "/fir/groups/bgirod/action_recognition/frames/v_ApplyEyeMakeup_g17_c03"<br/> Could you please collect the similar debug logs during migration?<br/> Thanks!</p> <p>Hi Hongchao,</p> <p>Since my last message, we have upgraded to 2.12.5 and I cannot reproduce the problem with the empty directory. It has now been successfully migrated to MDT1.</p> <p>However, we still have issues with EPERM errors even in 2.12.5.</p> <p>For example, I tried again today, and it still doesn't work for this directory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh02 ~]# lfs getdirstripe /fir/groups/astraigh/kousik lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64,migrating,lost_lmv mdtidx FID[seq:oid:ver] 0 [0x200042f8e:0x29:0x0] 3 [0x2800393f0:0x417d:0x0] [root@fir-rbh02 ~]# lfs migrate -m 3 /fir/groups/astraigh/kousik /fir/groups/astraigh/kousik migrate failed: Operation not permitted (-1) </pre> </div></div> <p>It looks like you spotted the problem (a previous migration was running). Is there a way to fix the problem so that we can migrate this directory to MDT3 for example?</p> <p>Thanks!<br/> Stephane</p> Related LU-13298 LU-13425 LU-15001 Development Rank 1|i00z3r: Rank (Obsolete) 9223372036854775807 Severity