[LU-6229] test racer with lustre_rsync

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.8.0
• Labels: patch
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: wu libin (Inactive)

Description

As i describe in LU-6166, the lustre_rsync has some problems, here is test result with racer, the result is:
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /home/target
Statuslog: /home/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 0
Clear changelog after use: no
Errors: 1057
lustre_rsync took 36 seconds
Changelog records consumed: 6774
Only in /mnt/lustre: .lustre
Only in /home/target: .lustrerepl
Only in /mnt/lustre/racer: 0
diff: /mnt/lustre/racer/1: Too many levels of symbolic links
Only in /home/target/racer: 10
diff: /mnt/lustre/racer/11/12: Too many levels of symbolic links
diff: /home/target/racer/11/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/13: Too many levels of symbolic links
diff: /home/target/racer/11/13: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/1: Too many levels of symbolic links
diff: /home/target/racer/11/16/1: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/10/0: No such file or directory
diff: /home/target/racer/11/16/10/0: No such file or directory
Only in /mnt/lustre/racer/11/16/10: 13
diff: /mnt/lustre/racer/11/16/10/18: Not a directory
diff: /home/target/racer/11/16/10/18: Not a directory
diff: /mnt/lustre/racer/11/16/10/6: Too many levels of symbolic links
diff: /home/target/racer/11/16/10/6: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 12
diff: /mnt/lustre/racer/11/16/13: Too many levels of symbolic links
diff: /home/target/racer/11/16/13: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 15
diff: /mnt/lustre/racer/11/16/17: Too many levels of symbolic links
diff: /home/target/racer/11/16/17: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 18
Only in /mnt/lustre/racer/11/16: 19
diff: /mnt/lustre/racer/11/16/2: Too many levels of symbolic links
diff: /home/target/racer/11/16/2: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/3: No such file or directory
diff: /home/target/racer/11/16/3: No such file or directory
diff: /mnt/lustre/racer/11/16/4: Too many levels of symbolic links
diff: /home/target/racer/11/16/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 5
Only in /mnt/lustre/racer/11/16: 6
diff: /mnt/lustre/racer/11/16/7/12: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/19: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/19: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16/7/8: 0
diff: /mnt/lustre/racer/11/16/7/8/10: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/8/16: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/16: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16/7/8: 2
Only in /mnt/lustre/racer/11/16/7/8/4: 11
Only in /mnt/lustre/racer/11/16/7/8/4: 19
diff: /mnt/lustre/racer/11/16/7/8/4/6: No such file or directory
diff: /home/target/racer/11/16/7/8/4/6: No such file or directory
diff: /mnt/lustre/racer/11/16/7/8/5: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/5: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/8/7: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/7: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/8: Too many levels of symbolic links
diff: /home/target/racer/11/16/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 9
diff: /mnt/lustre/racer/11/19: Too many levels of symbolic links
diff: /home/target/racer/11/19: Too many levels of symbolic links
diff: /home/target/racer/12: No such file or directory
Only in /mnt/lustre/racer/14: 0
Only in /mnt/lustre/racer/14: 1
Only in /mnt/lustre/racer/14: 10
Only in /mnt/lustre/racer/14: 11
Only in /mnt/lustre/racer/14/12: 15
diff: /mnt/lustre/racer/14/12/3: Too many levels of symbolic links
diff: /home/target/racer/14/12/3: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/12/5: Too many levels of symbolic links
diff: /home/target/racer/14/12/5: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/12: 6
diff: /mnt/lustre/racer/14/12/8: Too many levels of symbolic links
diff: /home/target/racer/14/12/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/14: 14
Only in /mnt/lustre/racer/14/16: 0
Only in /mnt/lustre/racer/14/16: 1
diff: /mnt/lustre/racer/14/16/6: Too many levels of symbolic links
diff: /home/target/racer/14/16/6: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/17: Too many levels of symbolic links
diff: /home/target/racer/14/17: Too many levels of symbolic links
Only in /mnt/lustre/racer/14: 18
Only in /mnt/lustre/racer/14/3: 1
Only in /mnt/lustre/racer/14/3: 10
diff: /mnt/lustre/racer/14/3/11: Too many levels of symbolic links
diff: /home/target/racer/14/3/11: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/3: 12
Only in /mnt/lustre/racer/14/3: 13
Only in /mnt/lustre/racer/14/3: 14
Only in /mnt/lustre/racer/14/3: 16
Only in /mnt/lustre/racer/14/3: 17
diff: /mnt/lustre/racer/14/3/19: Too many levels of symbolic links
diff: /home/target/racer/14/3/19: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/3: 2
Only in /mnt/lustre/racer/14/3: 3
Only in /mnt/lustre/racer/14/3: 4
Only in /mnt/lustre/racer/14/3: 7
Only in /mnt/lustre/racer/14/3: 8
Only in /mnt/lustre/racer/14/3: 9
Only in /mnt/lustre/racer/14: 4
Only in /mnt/lustre/racer/14/5: 0
Only in /mnt/lustre/racer/14/5: 1
Only in /mnt/lustre/racer/14/5: 11
Only in /mnt/lustre/racer/14/5: 12
diff: /mnt/lustre/racer/14/5/13: Too many levels of symbolic links
diff: /home/target/racer/14/5/13: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/5: 14
File /mnt/lustre/racer/14/5/16 is a directory while file /home/target/racer/14/5/16 is a regular empty file
diff: /mnt/lustre/racer/14/5/3: Too many levels of symbolic links
diff: /home/target/racer/14/5/3: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/5: 6
Only in /mnt/lustre/racer/14/5: 7
Only in /mnt/lustre/racer/14/5: 9
Only in /mnt/lustre/racer/14/6: 1
Only in /mnt/lustre/racer/14/6: 12
Only in /mnt/lustre/racer/14/6: 14
Only in /mnt/lustre/racer/14/6: 16
diff: /mnt/lustre/racer/14/6/18/0: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/0: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/10: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 11
diff: /mnt/lustre/racer/14/6/18/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/11: No such file or directory
diff: /home/target/racer/14/6/18/13/11: No such file or directory
diff: /mnt/lustre/racer/14/6/18/13/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/15: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/15: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 17
diff: /mnt/lustre/racer/14/6/18/13/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/18: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/4: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 5
diff: /mnt/lustre/racer/14/6/18/13/7: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/7: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 8
Only in /mnt/lustre/racer/14/6/18/13: 9
Only in /mnt/lustre/racer/14/6/18: 16
diff: /mnt/lustre/racer/14/6/18/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 19
Only in /mnt/lustre/racer/14/6/18: 2
Only in /mnt/lustre/racer/14/6/18: 3
diff: /mnt/lustre/racer/14/6/18/4: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 5
diff: /mnt/lustre/racer/14/6/18/7/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/10: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7: 12
Only in /mnt/lustre/racer/14/6/18/7: 14
Only in /mnt/lustre/racer/14/6/18/7/15: 0
Only in /mnt/lustre/racer/14/6/18/7/15: 1
diff: /mnt/lustre/racer/14/6/18/7/15/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/11: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/11: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/12: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7/15/13: 14
Only in /mnt/lustre/racer/14/6/18/7/15/13: 19
Only in /mnt/lustre/racer/14/6/18/7/15/13: 2
Only in /mnt/lustre/racer/14/6/18/7/15/13: 4
Only in /mnt/lustre/racer/14/6/18/7/15/13: 5
Only in /mnt/lustre/racer/14/6/18/7/15/13: 6
Only in /mnt/lustre/racer/14/6/18/7/15/13: 7
Only in /mnt/lustre/racer/14/6/18/7/15/13: 8
Only in /mnt/lustre/racer/14/6/18/7/15/13: 9
Only in /mnt/lustre/racer/14/6/18/7/15: 14
Only in /mnt/lustre/racer/14/6/18/7/15: 15
Only in /mnt/lustre/racer/14/6/18/7/15: 16
diff: /mnt/lustre/racer/14/6/18/7/15/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7/15: 3
diff: /mnt/lustre/racer/14/6/18/7/15/4: No such file or directory
diff: /mnt/lustre/racer/14/6/18/7/15/6: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/6: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/7: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/7: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/8: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/8: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/9: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/9: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7: 19
Only in /mnt/lustre/racer/14/6/18/7: 2
Only in /mnt/lustre/racer/14/6/18/7: 6
Only in /mnt/lustre/racer/14/6/18/7/7: 0
Only in /mnt/lustre/racer/14/6/18/7/7: 11
Only in /mnt/lustre/racer/14/6/18/7/7: 17
Only in /mnt/lustre/racer/14/6/18/7/7: 2
Only in /mnt/lustre/racer/14/6/18/7/7: 6
Only in /mnt/lustre/racer/14/6/18/7: 9
diff: /mnt/lustre/racer/14/6/18/8: No such file or directory
diff: /home/target/racer/14/6/18/8: No such file or directory
Only in /mnt/lustre/racer/14/6/18/9: 0
Only in /mnt/lustre/racer/14/6/18/9: 10
Only in /mnt/lustre/racer/14/6/18/9: 11
Only in /mnt/lustre/racer/14/6/18/9: 15
Only in /mnt/lustre/racer/14/6/18/9: 16
Only in /mnt/lustre/racer/14/6/18/9: 19
Only in /mnt/lustre/racer/14/6/18/9: 3
Only in /mnt/lustre/racer/14/6: 3
Only in /mnt/lustre/racer/14: 7
Only in /mnt/lustre/racer/14/8: 1
diff: /mnt/lustre/racer/14/8/10: Too many levels of symbolic links
diff: /home/target/racer/14/8/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/8/11/8: No such file or directory
diff: /home/target/racer/14/8/11/8: No such file or directory
Only in /mnt/lustre/racer/14/8/11: 9
Only in /mnt/lustre/racer/14/8: 13
Only in /mnt/lustre/racer/14/8: 14
diff: /mnt/lustre/racer/14/8/15: Too many levels of symbolic links
diff: /home/target/racer/14/8/15: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8: 16
Only in /mnt/lustre/racer/14/8: 17
Only in /mnt/lustre/racer/14/8: 18
Only in /mnt/lustre/racer/14/8: 19
diff: /mnt/lustre/racer/14/8/4/0: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/0: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 1
Only in /mnt/lustre/racer/14/8/4: 10
diff: /mnt/lustre/racer/14/8/4/11: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/11: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 12
diff: /mnt/lustre/racer/14/8/4/13: No such file or directory
Only in /mnt/lustre/racer/14/8/4: 15
Only in /mnt/lustre/racer/14/8/4: 16
diff: /mnt/lustre/racer/14/8/4/17: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/17: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/8/4/18: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 19
Only in /mnt/lustre/racer/14/8/4: 2
Only in /mnt/lustre/racer/14/8/4: 3
diff: /mnt/lustre/racer/14/8/4/5: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/5: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 6
Only in /mnt/lustre/racer/14/8/4: 8
Only in /mnt/lustre/racer/14/8/4: 9
Only in /mnt/lustre/racer/14/8: 6
Only in /mnt/lustre/racer/14/8: 7
Only in /mnt/lustre/racer/14: 9
diff: /home/target/racer/15: No such file or directory
Only in /mnt/lustre/racer: 16
Only in /mnt/lustre/racer/17: 0
Only in /mnt/lustre/racer/17: 1
Only in /mnt/lustre/racer/17: 10
Only in /mnt/lustre/racer/17: 13
Only in /mnt/lustre/racer/17: 16
Only in /mnt/lustre/racer/17: 17
diff: /mnt/lustre/racer/17/18: No such file or directory
Only in /mnt/lustre/racer/17: 19
diff: /mnt/lustre/racer/17/2: No such file or directory
diff: /home/target/racer/17/2: No such file or directory
Only in /home/target/racer/17: 4
Only in /mnt/lustre/racer/17: 5
Only in /mnt/lustre/racer/17: 7
Only in /mnt/lustre/racer/17: 8
Only in /mnt/lustre/racer/17: 9
Only in /mnt/lustre/racer/18: 1
Only in /mnt/lustre/racer/18: 12
Only in /mnt/lustre/racer/18: 16
Only in /home/target/racer/18: 2
diff: /mnt/lustre/racer/18/4: Too many levels of symbolic links
diff: /home/target/racer/18/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/18: 8
Only in /home/target/racer: 2
Only in /mnt/lustre/racer/3: 1
Only in /mnt/lustre/racer/3: 12
Only in /mnt/lustre/racer/3: 16
Only in /home/target/racer/3: 2
diff: /mnt/lustre/racer/3/4: Too many levels of symbolic links
diff: /home/target/racer/3/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/3: 8
diff: /mnt/lustre/racer/4/10: Too many levels of symbolic links
diff: /home/target/racer/4/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/4/7: No such file or directory
diff: /home/target/racer/4/7: No such file or directory
diff: /mnt/lustre/racer/5: Too many levels of symbolic links
diff: /mnt/lustre/racer/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/9: 15
diff: /mnt/lustre/racer/9/2: Too many levels of symbolic links
diff: /home/target/racer/9/2: Too many levels of symbolic links
diff: /mnt/lustre/racer/9/6: Too many levels of symbolic links
diff: /home/target/racer/9/6: Too many levels of symbolic links
lustre-rsync-bug test_13: @@@@@@ FAIL: Failure in replication; differences found.
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:4343:error_noexit()
= /usr/lib64/lustre/tests/test-framework.sh:4374:error()
= lustre-rsync-bug.sh:144:check_diff()
= lustre-rsync-bug.sh:203:test_13()
= /usr/lib64/lustre/tests/test-framework.sh:4613:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:4648:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:4516:run_test()
= lustre-rsync-bug.sh:210:main()
Dumping lctl log to /home/test_logs/1423558610/lustre-rsync-bug.test_13.*.1423558690.log

The attached file is the test script.