https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11753 Lustre <p>Client doing a migration of a directory from mdt0 to mdt1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ cd /fir/users/sthiell/mdtest $ lfs migrate -m 1 32142854 </pre> </div></div> <p>MDT0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[153404.321665] BUG: unable to handle kernel paging request at ffff903cb4a7d000 [153404.328760] IP: [&lt;ffffffffb9d7f29e&gt;] strncpy+0x1e/0x30 [153404.334023] PGD 6ae452067 PUD 2038b43063 PMD 20386e0063 PTE 8000002034a7d061 [153404.341259] Oops: 0003 [#1] SMP [153404.344632] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser ib_umad rdma_cm iw_cm libiscsi ib_ipoib scsi_transport_iscsi ib_cm mlx5_ib ib_core mpt2sas mptctl mptbase dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses dm_multipath enclosure ipmi_si pcspkr ipmi_devintf dm_mod ccp sg ipmi_msghandler i2c_piix4 k10temp acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif [153404.417761] crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core ttm ahci mlxfw libahci drm devlink crct10dif_pclmul crct10dif_common tg3 crc32c_intel ptp megaraid_sas libata drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas [153404.443948] CPU: 5 PID: 45301 Comm: mdt_out01_004 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [153404.456538] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.3.6 04/20/2018 [153404.464191] task: ffff902c9aa3b0c0 ti: ffff903cb5bec000 task.ti: ffff903cb5bec000 [153404.471757] RIP: 0010:[&lt;ffffffffb9d7f29e&gt;] [&lt;ffffffffb9d7f29e&gt;] strncpy+0x1e/0x30 [153404.479435] RSP: 0018:ffff903cb5befaf0 EFLAGS: 00010206 [153404.484835] RAX: ffff903cb4a7d000 RBX: ffff903cb4a7cfe0 RCX: ffff903cb4a7d000 [153404.492053] RDX: 0000000000000064 RSI: ffff903ca7d1213e RDI: ffff903cb4a7d000 [153404.499275] RBP: ffff903cb5befaf0 R08: ffff903cb4a7d010 R09: 0000000000000018 [153404.506495] R10: ffff902cbca82400 R11: ffff902cbca82400 R12: ffff901de9582000 [153404.513714] R13: ffff903ca7d12118 R14: 0000000000000000 R15: 0000000000000010 [153404.520933] FS: 00007ff8bee70740(0000) GS:ffff903cbf640000(0000) knlGS:0000000000000000 [153404.529105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [153404.534939] CR2: ffff903cb4a7d000 CR3: 00000006ade10000 CR4: 00000000003407e0 [153404.542157] Call Trace: [153404.544711] [&lt;ffffffffc118a6ac&gt;] osd_it_ea_rec+0x2ec/0x610 [osd_ldiskfs] [153404.551605] [&lt;ffffffffc0c334b9&gt;] dt_index_page_build+0x149/0x470 [obdclass] [153404.558756] [&lt;ffffffffc0c330e0&gt;] dt_index_walk+0x1a0/0x430 [obdclass] [153404.565386] [&lt;ffffffffc0c33370&gt;] ? dt_index_walk+0x430/0x430 [obdclass] [153404.572190] [&lt;ffffffffc0c34444&gt;] dt_index_read+0x394/0x6a0 [obdclass] [153404.578848] [&lt;ffffffffc0eceb32&gt;] tgt_obd_idx_read+0x612/0x860 [ptlrpc] [153404.585579] [&lt;ffffffffc0ed135a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [153404.592570] [&lt;ffffffffc0eaaa51&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [153404.600233] [&lt;ffffffffc0aacbde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [153404.607399] [&lt;ffffffffc0e7592b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [153404.615171] [&lt;ffffffffc0e727b5&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [153404.622046] [&lt;ffffffffb9ad67c2&gt;] ? default_wake_function+0x12/0x20 [153404.628403] [&lt;ffffffffb9acba9b&gt;] ? __wake_up_common+0x5b/0x90 [153404.634352] [&lt;ffffffffc0e7925c&gt;] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [153404.640729] [&lt;ffffffffc0e78760&gt;] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [153404.648208] [&lt;ffffffffb9ac1c31&gt;] kthread+0xd1/0xe0 [153404.653172] [&lt;ffffffffb9ac1b60&gt;] ? insert_kthread_work+0x40/0x40 [153404.659357] [&lt;ffffffffba174c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [153404.665886] [&lt;ffffffffb9ac1b60&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>Lustre 2.12.0 RC2</p> <p>Thanks,<br/> Stephane</p> CentOS 7.6 clients and servers (kernel 3.10.0-957.1.3.el7_lustre.x86_64) LU-11753

MDS BUG on lfs migrate [osd_it_ea_rec]

Bug Blocker Resolved Fixed Lai Siyao Stephane Thiell Mon, 10 Dec 2018 21:40:01 +0000 Wed, 6 Mar 2019 05:41:02 +0000 Mon, 17 Dec 2018 19:53:03 +0000 Lustre 2.12.0 Lustre 2.12.0 0 6 <p>Lai, could you please take a look?</p> <p>Stephane, is there anything particular about the <tt>mdtest</tt> directory that might trigger this (number of files, striping, etc)?</p> <p>Hey Andreas,</p> <p>The <tt>mdtest</tt> directory contains only a few sub directories (shown below), but the whole tree has 298,653 files.</p> <p>Actually, I now noticed that <tt>lmv_stripe_offset</tt> for <tt>32142854</tt> is 1, and the directory is already stripped over 2 MDTs. Unfortunately I don't have the output of getdirstripe before the lfs migrate sorry... The <tt>lmv_hash_type: -2147483646</tt> looks suspicious too. Please note that this test filesystem was formatted using 2.11.x with an old version of e2fsprogs (1.42.13.wc6-7).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -lisa mdtest/ total 48 144115205389956034 4 drwxr-xr-x 10 282232 32264 4096 Dec 10 13:22 . 144115205322833924 4 drwx--x--x 20 282232 32264 4096 Nov 29 14:09 .. 144115205591296683 4 drwxr-xr-x 3 282232 32264 4096 Nov 16 14:57 32141139 144115205591311199 4 drwxr-xr-x 6 282232 32264 4096 Nov 16 15:12 32141361 144115205591385583 4 drwxr-xr-x 4 282232 32264 4096 Nov 16 15:12 32142393 1026820799345983489 8 drwxr-xr-x 4 282232 32264 8192 Nov 16 15:19 32142698 162129637319639041 8 drwxr-xr-x 6 282232 32264 8192 Dec 10 13:22 32142854 144115205608131632 4 drwxr-xr-x 6 282232 32264 4096 Nov 16 15:48 32143730 144115205591391332 4 drwxr-xr-x 2 282232 32264 4096 Nov 16 15:16 3333 144115205591391333 4 drwxr-xr-x 2 282232 32264 4096 Nov 16 15:16 test # lfs getdirstripe mdtest lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none # for d in mdtest/*; do echo -n "$d: "; lfs getdirstripe $d; done mdtest/32141139: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtest/32141361: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtest/32142393: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtest/32142698: lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: -2147483646 mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x1:0x0] 0 [0x200000414:0x1dc66:0x0] mdtest/32142854: lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: -2147483646 mdtidx FID[seq:oid:ver] 1 [0x240000bd1:0x1:0x0] 0 [0x200000414:0x1ef5f:0x0] mdtest/32143730: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtest/3333: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none mdtest/test: lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none </pre> </div></div> <p>The <tt>lmv_hash_type</tt> printing is just a minor bug in <tt>lfs getdirstripe</tt> - it should be printing the hash type as an unsigned hex value <tt>0x80000002</tt>. There is a special flag (<tt>#define LMV_HASH_FLAG_MIGRATION 0x80000000</tt>) set on migrating directories to indicate that clients may need to check multiple MDTs to find filenames that are in the middle of migration.</p> <p>It looks that there is bug in readdir code between MDTs, normally this code is not called, so this bug was not seen before, but dir migration needs to readdir from remote MDT, which triggers this BUG, I'm reviewing related code and working on the fix.</p> <p>can you use 'echo t &gt; /proc/sysrq-triggers' to dump all processes backtrace on MDT1? this crash should have happened on MDT0, right?</p> <p>Hi Lai,</p> <p>Yes it happened on MDT0 (on server <tt>fir-md1-s1</tt> which is running MDT0 and MDT2). The other server <tt>fir-md1-s2</tt> is running MDT1 and MDT3 and I'm attaching the output of sysrq t there, however, please note that since the crash, I have restarted <tt>fir-md1-s1</tt> and MDT0,2. If you want the dump just after the crash, I can retry to trigger it tomorrow.</p> <p>Here is another occurence of this issue when doing a <tt>lfs migrate -m 1</tt> of a copy of the directory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15418.196245] BUG: unable to handle kernel paging request at ffff91c0f5f44014 [15418.203259] IP: [&lt;ffffffffc1469665&gt;] osd_it_ea_rec+0x2a5/0x610 [osd_ldiskfs] [15418.210349] PGD 1842e52067 PUD 203abc5063 PMD 2038f75063 PTE 8000002035f44061 [15418.217585] Oops: 0003 [#1] SMP [15418.220872] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser ib_umad rdma_cm iw_cm libiscsi ib_ipoib scsi_transport_iscsi ib_cm mlx5_ib ib_core mpt2sas mptctl mptbase dell_rbu sunrpc vfat fat dm_round_robin dcdbas amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses dm_multipath enclosure pcspkr ipmi_si dm_mod ccp ipmi_devintf k10temp sg i2c_piix4 ipmi_msghandler acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif [15418.293914] crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect ahci sysimgblt mlx5_core crct10dif_pclmul fb_sys_fops crct10dif_common libahci tg3 ttm mlxfw devlink drm crc32c_intel libata ptp drm_panel_orientation_quirks megaraid_sas pps_core mpt3sas(OE) raid_class scsi_transport_sas [15418.320099] CPU: 21 PID: 21111 Comm: mdt_out01_001 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [15418.332688] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.3.6 04/20/2018 [15418.340256] task: ffff91c0f9ff8000 ti: ffff91c0f6cbc000 task.ti: ffff91c0f6cbc000 [15418.347733] RIP: 0010:[&lt;ffffffffc1469665&gt;] [&lt;ffffffffc1469665&gt;] osd_it_ea_rec+0x2a5/0x610 [osd_ldiskfs] [15418.357242] RSP: 0018:ffff91c0f6cbfb00 EFLAGS: 00010246 [15418.362553] RAX: 0ba8ff12f067836e RBX: ffff91c0f5f43ff8 RCX: ffff91c0f98fedc8 [15418.369688] RDX: 0000000000000005 RSI: ffff91c0f98fedee RDI: 0000000000000000 [15418.376820] RBP: ffff91c0f6cbfb58 R08: 0000000200004277 R09: 0000000000000018 [15418.383951] R10: ffff91b036e2c700 R11: ffff91b036e2c700 R12: ffff91a2295c6000 [15418.391085] R13: ffff91c0f98fedc8 R14: 0000000000000000 R15: 0000000000000011 [15418.398219] FS: 00007f4d40fc6880(0000) GS:ffff91c0ff740000(0000) knlGS:0000000000000000 [15418.406304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15418.412049] CR2: ffff91c0f5f44014 CR3: 000000203a298000 CR4: 00000000003407e0 [15418.419182] Call Trace: [15418.421662] [&lt;ffffffffc0d104b9&gt;] dt_index_page_build+0x149/0x470 [obdclass] [15418.428727] [&lt;ffffffffc0d100e0&gt;] dt_index_walk+0x1a0/0x430 [obdclass] [15418.435270] [&lt;ffffffffc0d10370&gt;] ? dt_index_walk+0x430/0x430 [obdclass] [15418.441986] [&lt;ffffffffc0d11444&gt;] dt_index_read+0x394/0x6a0 [obdclass] [15418.448557] [&lt;ffffffffc0fabb32&gt;] tgt_obd_idx_read+0x612/0x860 [ptlrpc] [15418.455203] [&lt;ffffffffc0fae35a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [15418.462108] [&lt;ffffffffc0f87a51&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [15418.469681] [&lt;ffffffffc0ba2bde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [15418.476761] [&lt;ffffffffc0f5292b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [15418.484439] [&lt;ffffffffc0f4f7b5&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [15418.491224] [&lt;ffffffff9a4d67c2&gt;] ? default_wake_function+0x12/0x20 [15418.497487] [&lt;ffffffff9a4cba9b&gt;] ? __wake_up_common+0x5b/0x90 [15418.503350] [&lt;ffffffffc0f5625c&gt;] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [15418.509641] [&lt;ffffffffc0f55760&gt;] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [15418.517033] [&lt;ffffffff9a4c1c31&gt;] kthread+0xd1/0xe0 [15418.521911] [&lt;ffffffff9a4c1b60&gt;] ? insert_kthread_work+0x40/0x40 [15418.528006] [&lt;ffffffff9ab74c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [15418.534442] [&lt;ffffffff9a4c1b60&gt;] ? insert_kthread_work+0x40/0x40 [15418.540533] Code: 00 00 00 00 00 00 45 31 f6 49 8b 8f 10 01 00 00 8b 41 22 89 fa 44 0f b7 79 20 83 ca 01 83 e7 02 48 8d 71 26 89 45 c0 48 8b 41 18 &lt;89&gt; 53 1c 49 8b 55 00 48 89 13 41 8b 55 08 89 53 08 41 8b 55 0c [15418.561057] RIP [&lt;ffffffffc1469665&gt;] osd_it_ea_rec+0x2a5/0x610 [osd_ldiskfs] [15418.568223] RSP &lt;ffff91c0f6cbfb00&gt; [15418.571716] CR2: ffff91c0f5f44014 </pre> </div></div> <p> <br/> MDT0 crashed</p> <p>commands on clients:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 mdtest]# cp -r mdtest/32142854 mdtest/32142854-copy cp: cannot create directory ‘mdtest/32142854-copy/#test-dir.3’: File exists cp: cannot create directory ‘mdtest/32142854-copy/#test-dir.2/mdtest_tree.0’: File exists [root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@fir-rbh01 mdtest]# lfs migrate -m 1 32142854-copy &lt;mdt0 crash&gt; </pre> </div></div> <p>vmcore (mdt0) uploaded to the ftp as <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a>-vmcore-fir-md1-s1-2018-12-11-14-48-11<br/> debuginfo rpms available there too</p> <p>output of sysrq trigger "t" on mdt1 (<tt>fir-md1-s2</tt>) while <tt>fir-md1-s1</tt> was NOT restarted yet have been attached as fir-md1-s2_sysrq_t_2.log</p> <p>Thanks, I'll check that, can you also post the script of the whole process, which can help to add a test for this.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33837" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33837</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> obdclass: index_page support variable length rec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7924513ad8ac82971b4d93914c1dc6d7b679610d</p> <p>Hi Lai,</p> <p>I just tried your patch on top of 2.12.0 RC2 and did a <tt>lfs migrate -m 1</tt> on a copy of the original directory, and a crash occurred again (sorry!). Please see attached file vmcore-dmesg-fir-md1-s1-2018-12-12-12_18_37.txt</p> <p>The whole process is that I recursively copied a directory where I previously generated a lot of files using mdtest. The directory is stripped on MDT0, then I just <tt>lfs migrate</tt> it to MDT1.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 mdtest]# lfs getdirstripe 32142854 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: -2147483646 mdtidx FID[seq:oid:ver]      1 [0x240000bd1:0x1:0x0]      0 [0x200000414:0x1ef5f:0x0] [root@fir-rbh01 mdtest]# cp -r 32142854 32142854-copy2 cp: cannot create directory ‘32142854-copy2/#test-dir.3’: File exists cp: cannot create directory ‘32142854-copy2/#test-dir.2/mdtest_tree.0’: File exists [root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy2 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@fir-rbh01 mdtest]# find 32142854-copy2 | wc -l 89297 [root@fir-rbh01 mdtest]# lfs migrate -m 1 32142854-copy2 </pre> </div></div> <p>Perhaps something is wrong with the original directory 32142854? It's possible that is is only partially migrated. But in any case, it would be nice to avoid the crash. Thanks!</p> <p>Is it possible that this is related to <a href="https://jira.whamcloud.com/browse/LU-11330" title="replay-single test_70d: Directory not empty rmdir fails" class="issue-link" data-issue-key="LU-11330"><del>LU-11330</del></a>, which causes errors like "<tt>rm: cannot remove '/mnt/lustre/d70d.replay-single/test1': Directory not empty</tt>"?</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33843" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33843</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> utils: print out DNE2 directory hash flags<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3e8cc55044cf097450493f0cf9bfded24f356ef7</p> <p>Stephane, can you apply the above patch on MDS and test again?</p> <p>Lai, I already tried with <a href="https://review.whamcloud.com/33837" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33837</a> (patchset 1), see my comment above.</p> <p>Stephane, can you tar the whole directory of 32142854-copy2 and upload it? I want to test with it myself.</p> <p>Lai, sure, this is done!</p> <p>What is strange are the errors reported during copy in Stephane's test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cp -r mdtest/32142854 mdtest/32142854-copy cp: cannot create directory ‘mdtest/32142854-copy/#test-dir.3’: File exists cp: cannot create directory ‘mdtest/32142854-copy/#test-dir.2/mdtest_tree.0’: File exists </pre> </div></div> <p>Is that because the <tt>mdtest/32142854-copy</tt> directory already exists <em>before</em> the copy and those directories already exist, or are these errors generated when copying to a new directory?</p> <p>Indeed that is strange. The destination directory doesn't exist before the copy. I noticed that these two directories have several Links, esp <tt>#test-dir.2/mdtest_tree.0</tt> which has many of them..</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 mdtest]# stat 32142854-copy/#test-dir.2/mdtest_tree.0 File: ‘32142854-copy/#test-dir.2/mdtest_tree.0’ Size: 700416 Blocks: 1368 IO Block: 4096 directory Device: e64e03a8h/3863872424d Inode: 144115473741577975 Links: 4008 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-12-11 14:43:42.000000000 -0800 Modify: 2018-12-11 14:44:40.000000000 -0800 Change: 2018-12-11 14:44:40.000000000 -0800 Birth: - [root@fir-rbh01 mdtest]# stat 32142854-copy/#test-dir.3 File: ‘32142854-copy/#test-dir.3’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: e64e03a8h/3863872424d Inode: 144115473741512706 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-12-11 14:39:36.000000000 -0800 Modify: 2018-12-11 14:39:36.000000000 -0800 Change: 2018-12-11 14:39:36.000000000 -0800 Birth: - </pre> </div></div> <p>Never mind about the link count, it's just that this directory has a lot of sub directories.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33865" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33865</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> obdclass: lu_dirent record length missing '0'<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aea1bb0102fb7eb23e5df592ab401f470faef27c</p> <p>Stephane, I uploaded another patch, can you apply both and test again?</p> <p>Lai,<br/> Good news, MDS is not crashing anymore and <tt>lfs migrate</tt> worked!!</p> <p>Version tested (servers and client): 2.12.0 RC2 + 3 patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git log --oneline b3de10f LU-11753 obdclass: lu_dirent record length missing '0' 7386d0f LU-11753 utils: print out DNE2 directory hash flags 3a78e96 LU-11753 obdclass: index_page support variable length rec a63510c New release candidate 2.12.0-RC2 ... </pre> </div></div> <p>Results (no crash!):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 mdtest]# cp -r 32142854 32142854-copy4 cp: cannot create directory ‘32142854-copy4/#test-dir.3’: File exists cp: cannot create directory ‘32142854-copy4/#test-dir.2/mdtest_tree.0’: File exists [root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy4 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@fir-rbh01 mdtest]# lfs migrate -m 1 32142854-copy4 [root@fir-rbh01 mdtest]# [root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy4 lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none </pre> </div></div> <p>=&gt; Success <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p> </p> <p>Interestingly, during <tt>lfs migrate</tt>, the dir is stripped on the 2 MDTs (source and destination) and seen as migrating, which seems perfectly normal:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 32142854-copy4]# lfs getdirstripe . lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: fnv_1a_64,migrating mdtidx FID[seq:oid:ver] 1 [0x240003ab0:0x3:0x0] 0 [0x200007161:0x1:0x0] </pre> </div></div> <p>Another minor thing to note, is that after the migration, I got the following error from another terminal with a CWD within the migrated directory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 32142854-copy4]# lfs getdirstripe . lfs getdirstripe: cannot open '.': No such file or directory (2) error: getdirstripe failed for .. </pre> </div></div> <p>This is probably a limitation of lfs migrate at the moment but I'm fine with that!</p> <p>Thanks much Lai!</p> <p>Hey, I just wanted to add that BOTH patches from Lai really seem to be needed to make the MDS crash go away and lfs migrate actually work. I already knew that the first one (<a href="https://review.whamcloud.com/33837" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33837</a>) alone was not enough, and so FYI, I just tried with only  this one <a href="https://review.whamcloud.com/33865" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33865</a> and got a crash in dt_index_page_build.</p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sthiell" class="user-hover" rel="sthiell">sthiell</a> could you please re-try with the latest versions of the patches - Alex made a tweak because of some issues that came up during regression testing....</p> <p>Peter, ok great, no problem! working on it now.</p> <p>Patch tested on top of 2.12.0 RC2:<br/> <a href="https://review.whamcloud.com/#/c/33837/4" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33837/4</a><br/> <a href="https://review.whamcloud.com/#/c/33865/1" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33865/1</a><br/>  <br/> Result: success!</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy6 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none [root@fir-rbh01 mdtest]# lfs migrate -m 1 32142854-copy6 [root@fir-rbh01 mdtest]# lfs getdirstripe 32142854-copy6 lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none </pre> </div></div> <p>BTW I believe the "File exists" errors are due to the fact that the original directory 32142854 is partially migrated and MDS crashed several times, so maybe the two directories are present in both MDT? I don't want to resume its migration now because the directory is part of my reproducer for the current issue, but we can probably confirm that later.</p> <p>Thank you all!<br/>  </p> <p>Fantastic news - thanks Stephane!</p> <blockquote> <p>I got the following error from another terminal with a CWD within the migrated directory:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-rbh01 32142854-copy4]# lfs getdirstripe . lfs getdirstripe: cannot open '.': No such file or directory (2) error: getdirstripe failed for .. This is probably a limitation of lfs migrate at the moment but I'm fine with that! </pre> </div></div></blockquote> <p>Yes, migrating files/directories between MDTs changes their FIDs, which is important to note if your HSM copytool depends on FIDs... There is <a href="https://jira.whamcloud.com/browse/LU-7607" title="Preserve inode number after MDT migration " class="issue-link" data-issue-key="LU-7607">LU-7607</a> "Preserve inode number after MDT migration" but it is not implemented yet.</p> <p>Ok I see, thanks Andreas! (the correct LU# seems to be <a href="https://jira.whamcloud.com/browse/LU-7607" title="Preserve inode number after MDT migration " class="issue-link" data-issue-key="LU-7607">LU-7607</a>)</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33843/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33843/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> utils: print out DNE2 directory hash flags<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 795eea12271f5c1bab3414803db3538d7c266d66</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33837/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33837/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> obdclass: index_page support variable length rec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a13e4991a1350f54f97c6ba13686d33c7a3eeb57</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33865/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33865/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> obdclass: lu_dirent record length missing '0'<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 77f01308c5095030ee84f83339c085bcbcf04155</p> <p>Landed for 2.12</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34376" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34376</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11753" title="MDS BUG on lfs migrate [osd_it_ea_rec]" class="issue-link" data-issue-key="LU-11753"><del>LU-11753</del></a> utils: print out DNE2 directory hash flags<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 47096b801de9b70f01caa7aae4104f2b851bc474</p> Duplicate LU-11781 Related LU-7607 Development Rank 1|i007q7: Rank (Obsolete) 9223372036854775807 Severity