https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6229 Lustre <p>As i describe in <a href="https://jira.whamcloud.com/browse/LU-6166" title="bugs of lustre_rsync" class="issue-link" data-issue-key="LU-6166"><del>LU-6166</del></a>, the lustre_rsync has some problems, here is test result with racer, the result is:<br/> Lustre filesystem: lustre<br/> MDT device: lustre-MDT0000<br/> Source: /mnt/lustre<br/> Target: /home/target<br/> Statuslog: /home/lustre_rsync.log<br/> Changelog registration: cl1<br/> Starting changelog record: 0<br/> Clear changelog after use: no<br/> Errors: 1057<br/> lustre_rsync took 36 seconds<br/> Changelog records consumed: 6774<br/> Only in /mnt/lustre: .lustre<br/> Only in /home/target: .lustrerepl<br/> Only in /mnt/lustre/racer: 0<br/> diff: /mnt/lustre/racer/1: Too many levels of symbolic links<br/> Only in /home/target/racer: 10<br/> diff: /mnt/lustre/racer/11/12: Too many levels of symbolic links<br/> diff: /home/target/racer/11/12: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/13: Too many levels of symbolic links<br/> diff: /home/target/racer/11/13: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/1: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/1: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/10/0: No such file or directory<br/> diff: /home/target/racer/11/16/10/0: No such file or directory<br/> Only in /mnt/lustre/racer/11/16/10: 13<br/> diff: /mnt/lustre/racer/11/16/10/18: Not a directory<br/> diff: /home/target/racer/11/16/10/18: Not a directory<br/> diff: /mnt/lustre/racer/11/16/10/6: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/10/6: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16: 12<br/> diff: /mnt/lustre/racer/11/16/13: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/13: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16: 15<br/> diff: /mnt/lustre/racer/11/16/17: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/17: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16: 18<br/> Only in /mnt/lustre/racer/11/16: 19<br/> diff: /mnt/lustre/racer/11/16/2: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/2: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/3: No such file or directory<br/> diff: /home/target/racer/11/16/3: No such file or directory<br/> diff: /mnt/lustre/racer/11/16/4: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/4: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16: 5<br/> Only in /mnt/lustre/racer/11/16: 6<br/> diff: /mnt/lustre/racer/11/16/7/12: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/12: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/7/19: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/19: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16/7/8: 0<br/> diff: /mnt/lustre/racer/11/16/7/8/10: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/8/10: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/7/8/16: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/8/16: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16/7/8: 2<br/> Only in /mnt/lustre/racer/11/16/7/8/4: 11<br/> Only in /mnt/lustre/racer/11/16/7/8/4: 19<br/> diff: /mnt/lustre/racer/11/16/7/8/4/6: No such file or directory<br/> diff: /home/target/racer/11/16/7/8/4/6: No such file or directory<br/> diff: /mnt/lustre/racer/11/16/7/8/5: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/8/5: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/7/8/7: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/7/8/7: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/11/16/8: Too many levels of symbolic links<br/> diff: /home/target/racer/11/16/8: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/11/16: 9<br/> diff: /mnt/lustre/racer/11/19: Too many levels of symbolic links<br/> diff: /home/target/racer/11/19: Too many levels of symbolic links<br/> diff: /home/target/racer/12: No such file or directory<br/> Only in /mnt/lustre/racer/14: 0<br/> Only in /mnt/lustre/racer/14: 1<br/> Only in /mnt/lustre/racer/14: 10<br/> Only in /mnt/lustre/racer/14: 11<br/> Only in /mnt/lustre/racer/14/12: 15<br/> diff: /mnt/lustre/racer/14/12/3: Too many levels of symbolic links<br/> diff: /home/target/racer/14/12/3: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/12/5: Too many levels of symbolic links<br/> diff: /home/target/racer/14/12/5: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/12: 6<br/> diff: /mnt/lustre/racer/14/12/8: Too many levels of symbolic links<br/> diff: /home/target/racer/14/12/8: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14: 14<br/> Only in /mnt/lustre/racer/14/16: 0<br/> Only in /mnt/lustre/racer/14/16: 1<br/> diff: /mnt/lustre/racer/14/16/6: Too many levels of symbolic links<br/> diff: /home/target/racer/14/16/6: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/17: Too many levels of symbolic links<br/> diff: /home/target/racer/14/17: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14: 18<br/> Only in /mnt/lustre/racer/14/3: 1<br/> Only in /mnt/lustre/racer/14/3: 10<br/> diff: /mnt/lustre/racer/14/3/11: Too many levels of symbolic links<br/> diff: /home/target/racer/14/3/11: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/3: 12<br/> Only in /mnt/lustre/racer/14/3: 13<br/> Only in /mnt/lustre/racer/14/3: 14<br/> Only in /mnt/lustre/racer/14/3: 16<br/> Only in /mnt/lustre/racer/14/3: 17<br/> diff: /mnt/lustre/racer/14/3/19: Too many levels of symbolic links<br/> diff: /home/target/racer/14/3/19: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/3: 2<br/> Only in /mnt/lustre/racer/14/3: 3<br/> Only in /mnt/lustre/racer/14/3: 4<br/> Only in /mnt/lustre/racer/14/3: 7<br/> Only in /mnt/lustre/racer/14/3: 8<br/> Only in /mnt/lustre/racer/14/3: 9<br/> Only in /mnt/lustre/racer/14: 4<br/> Only in /mnt/lustre/racer/14/5: 0<br/> Only in /mnt/lustre/racer/14/5: 1<br/> Only in /mnt/lustre/racer/14/5: 11<br/> Only in /mnt/lustre/racer/14/5: 12<br/> diff: /mnt/lustre/racer/14/5/13: Too many levels of symbolic links<br/> diff: /home/target/racer/14/5/13: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/5: 14<br/> File /mnt/lustre/racer/14/5/16 is a directory while file /home/target/racer/14/5/16 is a regular empty file<br/> diff: /mnt/lustre/racer/14/5/3: Too many levels of symbolic links<br/> diff: /home/target/racer/14/5/3: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/5: 6<br/> Only in /mnt/lustre/racer/14/5: 7<br/> Only in /mnt/lustre/racer/14/5: 9<br/> Only in /mnt/lustre/racer/14/6: 1<br/> Only in /mnt/lustre/racer/14/6: 12<br/> Only in /mnt/lustre/racer/14/6: 14<br/> Only in /mnt/lustre/racer/14/6: 16<br/> diff: /mnt/lustre/racer/14/6/18/0: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/0: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/10: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/10: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18: 11<br/> diff: /mnt/lustre/racer/14/6/18/12: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/12: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/13/11: No such file or directory<br/> diff: /home/target/racer/14/6/18/13/11: No such file or directory<br/> diff: /mnt/lustre/racer/14/6/18/13/12: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/13/12: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/13/15: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/13/15: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/13: 17<br/> diff: /mnt/lustre/racer/14/6/18/13/18: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/13/18: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/13/4: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/13/4: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/13: 5<br/> diff: /mnt/lustre/racer/14/6/18/13/7: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/13/7: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/13: 8<br/> Only in /mnt/lustre/racer/14/6/18/13: 9<br/> Only in /mnt/lustre/racer/14/6/18: 16<br/> diff: /mnt/lustre/racer/14/6/18/18: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/18: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18: 19<br/> Only in /mnt/lustre/racer/14/6/18: 2<br/> Only in /mnt/lustre/racer/14/6/18: 3<br/> diff: /mnt/lustre/racer/14/6/18/4: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/4: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18: 5<br/> diff: /mnt/lustre/racer/14/6/18/7/10: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/10: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/7: 12<br/> Only in /mnt/lustre/racer/14/6/18/7: 14<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 0<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 1<br/> diff: /mnt/lustre/racer/14/6/18/7/15/10: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/10: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/7/15/11: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/11: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/7/15/12: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/12: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 14<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 19<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 2<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 4<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 5<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 6<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 7<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 8<br/> Only in /mnt/lustre/racer/14/6/18/7/15/13: 9<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 14<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 15<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 16<br/> diff: /mnt/lustre/racer/14/6/18/7/15/18: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/18: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/7/15: 3<br/> diff: /mnt/lustre/racer/14/6/18/7/15/4: No such file or directory<br/> diff: /mnt/lustre/racer/14/6/18/7/15/6: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/6: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/7/15/7: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/7: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/7/15/8: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/8: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/6/18/7/15/9: Too many levels of symbolic links<br/> diff: /home/target/racer/14/6/18/7/15/9: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/6/18/7: 19<br/> Only in /mnt/lustre/racer/14/6/18/7: 2<br/> Only in /mnt/lustre/racer/14/6/18/7: 6<br/> Only in /mnt/lustre/racer/14/6/18/7/7: 0<br/> Only in /mnt/lustre/racer/14/6/18/7/7: 11<br/> Only in /mnt/lustre/racer/14/6/18/7/7: 17<br/> Only in /mnt/lustre/racer/14/6/18/7/7: 2<br/> Only in /mnt/lustre/racer/14/6/18/7/7: 6<br/> Only in /mnt/lustre/racer/14/6/18/7: 9<br/> diff: /mnt/lustre/racer/14/6/18/8: No such file or directory<br/> diff: /home/target/racer/14/6/18/8: No such file or directory<br/> Only in /mnt/lustre/racer/14/6/18/9: 0<br/> Only in /mnt/lustre/racer/14/6/18/9: 10<br/> Only in /mnt/lustre/racer/14/6/18/9: 11<br/> Only in /mnt/lustre/racer/14/6/18/9: 15<br/> Only in /mnt/lustre/racer/14/6/18/9: 16<br/> Only in /mnt/lustre/racer/14/6/18/9: 19<br/> Only in /mnt/lustre/racer/14/6/18/9: 3<br/> Only in /mnt/lustre/racer/14/6: 3<br/> Only in /mnt/lustre/racer/14: 7<br/> Only in /mnt/lustre/racer/14/8: 1<br/> diff: /mnt/lustre/racer/14/8/10: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/10: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/8/11/8: No such file or directory<br/> diff: /home/target/racer/14/8/11/8: No such file or directory<br/> Only in /mnt/lustre/racer/14/8/11: 9<br/> Only in /mnt/lustre/racer/14/8: 13<br/> Only in /mnt/lustre/racer/14/8: 14<br/> diff: /mnt/lustre/racer/14/8/15: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/15: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/8: 16<br/> Only in /mnt/lustre/racer/14/8: 17<br/> Only in /mnt/lustre/racer/14/8: 18<br/> Only in /mnt/lustre/racer/14/8: 19<br/> diff: /mnt/lustre/racer/14/8/4/0: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/4/0: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/8/4: 1<br/> Only in /mnt/lustre/racer/14/8/4: 10<br/> diff: /mnt/lustre/racer/14/8/4/11: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/4/11: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/8/4: 12<br/> diff: /mnt/lustre/racer/14/8/4/13: No such file or directory<br/> Only in /mnt/lustre/racer/14/8/4: 15<br/> Only in /mnt/lustre/racer/14/8/4: 16<br/> diff: /mnt/lustre/racer/14/8/4/17: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/4/17: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/14/8/4/18: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/4/18: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/8/4: 19<br/> Only in /mnt/lustre/racer/14/8/4: 2<br/> Only in /mnt/lustre/racer/14/8/4: 3<br/> diff: /mnt/lustre/racer/14/8/4/5: Too many levels of symbolic links<br/> diff: /home/target/racer/14/8/4/5: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/14/8/4: 6<br/> Only in /mnt/lustre/racer/14/8/4: 8<br/> Only in /mnt/lustre/racer/14/8/4: 9<br/> Only in /mnt/lustre/racer/14/8: 6<br/> Only in /mnt/lustre/racer/14/8: 7<br/> Only in /mnt/lustre/racer/14: 9<br/> diff: /home/target/racer/15: No such file or directory<br/> Only in /mnt/lustre/racer: 16<br/> Only in /mnt/lustre/racer/17: 0<br/> Only in /mnt/lustre/racer/17: 1<br/> Only in /mnt/lustre/racer/17: 10<br/> Only in /mnt/lustre/racer/17: 13<br/> Only in /mnt/lustre/racer/17: 16<br/> Only in /mnt/lustre/racer/17: 17<br/> diff: /mnt/lustre/racer/17/18: No such file or directory<br/> Only in /mnt/lustre/racer/17: 19<br/> diff: /mnt/lustre/racer/17/2: No such file or directory<br/> diff: /home/target/racer/17/2: No such file or directory<br/> Only in /home/target/racer/17: 4<br/> Only in /mnt/lustre/racer/17: 5<br/> Only in /mnt/lustre/racer/17: 7<br/> Only in /mnt/lustre/racer/17: 8<br/> Only in /mnt/lustre/racer/17: 9<br/> Only in /mnt/lustre/racer/18: 1<br/> Only in /mnt/lustre/racer/18: 12<br/> Only in /mnt/lustre/racer/18: 16<br/> Only in /home/target/racer/18: 2<br/> diff: /mnt/lustre/racer/18/4: Too many levels of symbolic links<br/> diff: /home/target/racer/18/4: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/18: 8<br/> Only in /home/target/racer: 2<br/> Only in /mnt/lustre/racer/3: 1<br/> Only in /mnt/lustre/racer/3: 12<br/> Only in /mnt/lustre/racer/3: 16<br/> Only in /home/target/racer/3: 2<br/> diff: /mnt/lustre/racer/3/4: Too many levels of symbolic links<br/> diff: /home/target/racer/3/4: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/3: 8<br/> diff: /mnt/lustre/racer/4/10: Too many levels of symbolic links<br/> diff: /home/target/racer/4/10: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/4/7: No such file or directory<br/> diff: /home/target/racer/4/7: No such file or directory<br/> diff: /mnt/lustre/racer/5: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/8: Too many levels of symbolic links<br/> Only in /mnt/lustre/racer/9: 15<br/> diff: /mnt/lustre/racer/9/2: Too many levels of symbolic links<br/> diff: /home/target/racer/9/2: Too many levels of symbolic links<br/> diff: /mnt/lustre/racer/9/6: Too many levels of symbolic links<br/> diff: /home/target/racer/9/6: Too many levels of symbolic links<br/> lustre-rsync-bug test_13: @@@@@@ FAIL: Failure in replication; differences found. <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:4343:error_noexit()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4374:error()<br/> = lustre-rsync-bug.sh:144:check_diff()<br/> = lustre-rsync-bug.sh:203:test_13()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4613:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4648:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4516:run_test()<br/> = lustre-rsync-bug.sh:210:main()<br/> Dumping lctl log to /home/test_logs/1423558610/lustre-rsync-bug.test_13.*.1423558690.log</p> <p>The attached file is the test script.</p> LU-6229

test racer with lustre_rsync

Bug Minor Resolved Fixed Niu Yawei wu libin patch Tue, 10 Feb 2015 09:17:38 +0000 Thu, 10 Dec 2015 18:04:32 +0000 Thu, 10 Dec 2015 18:04:32 +0000 Lustre 2.8.0 0 7 <p>Can you please elaborate on what is your test? Racer with lustre-rsync in parallel?</p> <p>The looping symlinks are probably to be expected and might be a test artifact where diff would try to follow symlinks.<br/> The missing files I am less sure about - do we have a hole in changelogs accounting somewhere? Could it be racer scripts are still running after rsync completed and kept making changes?</p> <p>Ah, my test is in the attached script(could you check the attached file?), it run like:<br/> 1. run racer for 10 second, stop<br/> 2. use lustre_rsync to sync the directory<br/> 3. use diff to check the difference</p> <p>I didn't find any hole in changelogs.</p> <p>Here i want to share a simple script will cause the problem:</p> <p>pushd /mnt/lustre<br/> 1. mkdir racer<br/> 2. mkdir racer/11<br/> 3. mkdir racer/13<br/> 4. mv racer/13 racer/14<br/> 5. mkdir racer/14/14/<br/> 6. mv racer/14 racer/11/<br/> popd</p> <p>After ran this script, use lustre_rsync to sync the data, process:<br/> 1. mkdir racer normal<br/> 2. mkdir racer/11 normal<br/> 3. mkdir racer/13, it's a special case, so mkdir under .lustrerepl directory<br/> 4. mv racer/13 racer/14, "racer/13" is a special case, find it under .lustrerepl and rename to "racer/14"<br/> 5. mkdir racer/14/14, it's a special case, mkdir under .lustrerepl directory<br/> 6. mv racer/14 racer/11, normal<br/> so, finally, "racer/14/14" is still under .lustrerepl, not under "racer/11/14/"</p> <p>That's what i said it could has orphan node under .lustrerepl.</p> <p>I see. thanks for these steps.<br/> Is this something you plan to fix yourself or do you need some help here?</p> <p>Yeah, i'm try to fix it myself, but i did not find the root cause, i'm glad that if someone here could help me.</p> <p>Following is how lustre_rsync works (Copied from the codes):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>* 1. creat * If tfid is absent on the source-fs, ignore this operation * If pfid is absent on the source-fs [or] * if f2p(pfid) is not present on target-fs [or] * if f2p(pfid)+name != f2p(tfid) * creat .lustrerepl/tfid * track [pfid,tfid,name] * Else * creat f2p[tfid] * * 2. remove * If .lustrerepl/[tfid] is present on the target * rm .lustrerepl/[tfid] * Else if pfid is present on the source-fs, * if f2p(pfid)+name is present, * rm f2p(pfid)+name * * 3. move (spfid,sname) to (pfid,name) * If pfid is present * if spfid is also present, mv (spfid,sname) to (pfid,name) * else mv .lustrerepl/[sfid] to (pfid,name) * Else if pfid is not present, * if spfid is present, mv (spfid,sname) .lustrerepl/[sfid] * move out all its children in .lustrerepl. * [pfid,tfid,name] tracked from (1) is used for this. </pre> </div></div> <p>And following is how the tree is created and replayed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1. mkdir racer racer 2. mkdir racer/11 racer/11 3. mkdir racer/13 racer/11 racer/13 4. mv race/13 racer/14 racer/11 race/14 5. mkdir racer/14/14 racer/11 race/14/14 6. mv racer/14 racer/11 racer/11/14/14 1. mkdir racer racer 2. mkdir racer/11 racer/11 3. mkdir racer/13 racer/11 .lustrerpl/tfidp[13] 4. mv racer/13 racer/14 racer/11 racer/14 5. mkdir racer/14/14 racer/11 racer/14 .lustrerpl/tfidp[14] 6. mv racer/14 racer/11 racer/11/14 .lustrerpl/tfidp[14] </pre> </div></div> <p>So it seems the step 4 of replay goes wrong already. Libin, are we able to reproduce with a single thread script?</p> <p>yeah, single thread of this script will cause the problem.</p> <p>Oleg, any suggestion about this problem?</p> <p>I might be wrong, but the following might be incorrect:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-xml">* 1. creat * If tfid is absent on the source-fs, ignore this operation * If pfid is absent on the source-fs [or] * if f2p(pfid) is not present on target-fs [or] * if f2p(pfid)+name != f2p(tfid) * creat .lustrerepl/tfid * track [pfid,tfid,name] * Else * creat f2p[tfid] </pre> </div></div> <p>In the case of 'if f2p(pfid)+name != f2p(tfid)', we need to create file tfid with path of "f2p(pfid)+name". I will make a patch and check whether this is the cause.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/14914" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14914</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6229" title="test racer with lustre_rsync" class="issue-link" data-issue-key="LU-6229"><del>LU-6229</del></a> utils: fix lustre_rsync bug of cascade move<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 055b39c94032bccbb6caf6f0308f08ae8dae64e3</p> <p>The script that can preproduce the problem easily.</p> <p>Thank you for the patch, LiXi.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14914/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14914/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6229" title="test racer with lustre_rsync" class="issue-link" data-issue-key="LU-6229"><del>LU-6229</del></a> utils: fix lustre_rsync bug of cascade move<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b9f3c954ef00c83ea34d0e3d7474f2b27eac29b8</p> <p>Landed for 2.8</p> Development Rank 1|hzx62f: Rank (Obsolete) 17439 Severity