[LU-6229] test racer with lustre_rsync

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.8.0
• Labels: patch
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: wu libin (Inactive)

Description

As i describe in LU-6166, the lustre_rsync has some problems, here is test result with racer, the result is:
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /home/target
Statuslog: /home/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 0
Clear changelog after use: no
Errors: 1057
lustre_rsync took 36 seconds
Changelog records consumed: 6774
Only in /mnt/lustre: .lustre
Only in /home/target: .lustrerepl
Only in /mnt/lustre/racer: 0
diff: /mnt/lustre/racer/1: Too many levels of symbolic links
Only in /home/target/racer: 10
diff: /mnt/lustre/racer/11/12: Too many levels of symbolic links
diff: /home/target/racer/11/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/13: Too many levels of symbolic links
diff: /home/target/racer/11/13: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/1: Too many levels of symbolic links
diff: /home/target/racer/11/16/1: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/10/0: No such file or directory
diff: /home/target/racer/11/16/10/0: No such file or directory
Only in /mnt/lustre/racer/11/16/10: 13
diff: /mnt/lustre/racer/11/16/10/18: Not a directory
diff: /home/target/racer/11/16/10/18: Not a directory
diff: /mnt/lustre/racer/11/16/10/6: Too many levels of symbolic links
diff: /home/target/racer/11/16/10/6: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 12
diff: /mnt/lustre/racer/11/16/13: Too many levels of symbolic links
diff: /home/target/racer/11/16/13: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 15
diff: /mnt/lustre/racer/11/16/17: Too many levels of symbolic links
diff: /home/target/racer/11/16/17: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 18
Only in /mnt/lustre/racer/11/16: 19
diff: /mnt/lustre/racer/11/16/2: Too many levels of symbolic links
diff: /home/target/racer/11/16/2: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/3: No such file or directory
diff: /home/target/racer/11/16/3: No such file or directory
diff: /mnt/lustre/racer/11/16/4: Too many levels of symbolic links
diff: /home/target/racer/11/16/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 5
Only in /mnt/lustre/racer/11/16: 6
diff: /mnt/lustre/racer/11/16/7/12: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/19: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/19: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16/7/8: 0
diff: /mnt/lustre/racer/11/16/7/8/10: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/8/16: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/16: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16/7/8: 2
Only in /mnt/lustre/racer/11/16/7/8/4: 11
Only in /mnt/lustre/racer/11/16/7/8/4: 19
diff: /mnt/lustre/racer/11/16/7/8/4/6: No such file or directory
diff: /home/target/racer/11/16/7/8/4/6: No such file or directory
diff: /mnt/lustre/racer/11/16/7/8/5: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/5: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/7/8/7: Too many levels of symbolic links
diff: /home/target/racer/11/16/7/8/7: Too many levels of symbolic links
diff: /mnt/lustre/racer/11/16/8: Too many levels of symbolic links
diff: /home/target/racer/11/16/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/11/16: 9
diff: /mnt/lustre/racer/11/19: Too many levels of symbolic links
diff: /home/target/racer/11/19: Too many levels of symbolic links
diff: /home/target/racer/12: No such file or directory
Only in /mnt/lustre/racer/14: 0
Only in /mnt/lustre/racer/14: 1
Only in /mnt/lustre/racer/14: 10
Only in /mnt/lustre/racer/14: 11
Only in /mnt/lustre/racer/14/12: 15
diff: /mnt/lustre/racer/14/12/3: Too many levels of symbolic links
diff: /home/target/racer/14/12/3: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/12/5: Too many levels of symbolic links
diff: /home/target/racer/14/12/5: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/12: 6
diff: /mnt/lustre/racer/14/12/8: Too many levels of symbolic links
diff: /home/target/racer/14/12/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/14: 14
Only in /mnt/lustre/racer/14/16: 0
Only in /mnt/lustre/racer/14/16: 1
diff: /mnt/lustre/racer/14/16/6: Too many levels of symbolic links
diff: /home/target/racer/14/16/6: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/17: Too many levels of symbolic links
diff: /home/target/racer/14/17: Too many levels of symbolic links
Only in /mnt/lustre/racer/14: 18
Only in /mnt/lustre/racer/14/3: 1
Only in /mnt/lustre/racer/14/3: 10
diff: /mnt/lustre/racer/14/3/11: Too many levels of symbolic links
diff: /home/target/racer/14/3/11: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/3: 12
Only in /mnt/lustre/racer/14/3: 13
Only in /mnt/lustre/racer/14/3: 14
Only in /mnt/lustre/racer/14/3: 16
Only in /mnt/lustre/racer/14/3: 17
diff: /mnt/lustre/racer/14/3/19: Too many levels of symbolic links
diff: /home/target/racer/14/3/19: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/3: 2
Only in /mnt/lustre/racer/14/3: 3
Only in /mnt/lustre/racer/14/3: 4
Only in /mnt/lustre/racer/14/3: 7
Only in /mnt/lustre/racer/14/3: 8
Only in /mnt/lustre/racer/14/3: 9
Only in /mnt/lustre/racer/14: 4
Only in /mnt/lustre/racer/14/5: 0
Only in /mnt/lustre/racer/14/5: 1
Only in /mnt/lustre/racer/14/5: 11
Only in /mnt/lustre/racer/14/5: 12
diff: /mnt/lustre/racer/14/5/13: Too many levels of symbolic links
diff: /home/target/racer/14/5/13: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/5: 14
File /mnt/lustre/racer/14/5/16 is a directory while file /home/target/racer/14/5/16 is a regular empty file
diff: /mnt/lustre/racer/14/5/3: Too many levels of symbolic links
diff: /home/target/racer/14/5/3: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/5: 6
Only in /mnt/lustre/racer/14/5: 7
Only in /mnt/lustre/racer/14/5: 9
Only in /mnt/lustre/racer/14/6: 1
Only in /mnt/lustre/racer/14/6: 12
Only in /mnt/lustre/racer/14/6: 14
Only in /mnt/lustre/racer/14/6: 16
diff: /mnt/lustre/racer/14/6/18/0: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/0: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/10: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 11
diff: /mnt/lustre/racer/14/6/18/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/11: No such file or directory
diff: /home/target/racer/14/6/18/13/11: No such file or directory
diff: /mnt/lustre/racer/14/6/18/13/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/12: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/15: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/15: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 17
diff: /mnt/lustre/racer/14/6/18/13/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/18: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/13/4: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 5
diff: /mnt/lustre/racer/14/6/18/13/7: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/13/7: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/13: 8
Only in /mnt/lustre/racer/14/6/18/13: 9
Only in /mnt/lustre/racer/14/6/18: 16
diff: /mnt/lustre/racer/14/6/18/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 19
Only in /mnt/lustre/racer/14/6/18: 2
Only in /mnt/lustre/racer/14/6/18: 3
diff: /mnt/lustre/racer/14/6/18/4: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18: 5
diff: /mnt/lustre/racer/14/6/18/7/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/10: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7: 12
Only in /mnt/lustre/racer/14/6/18/7: 14
Only in /mnt/lustre/racer/14/6/18/7/15: 0
Only in /mnt/lustre/racer/14/6/18/7/15: 1
diff: /mnt/lustre/racer/14/6/18/7/15/10: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/11: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/11: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/12: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/12: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7/15/13: 14
Only in /mnt/lustre/racer/14/6/18/7/15/13: 19
Only in /mnt/lustre/racer/14/6/18/7/15/13: 2
Only in /mnt/lustre/racer/14/6/18/7/15/13: 4
Only in /mnt/lustre/racer/14/6/18/7/15/13: 5
Only in /mnt/lustre/racer/14/6/18/7/15/13: 6
Only in /mnt/lustre/racer/14/6/18/7/15/13: 7
Only in /mnt/lustre/racer/14/6/18/7/15/13: 8
Only in /mnt/lustre/racer/14/6/18/7/15/13: 9
Only in /mnt/lustre/racer/14/6/18/7/15: 14
Only in /mnt/lustre/racer/14/6/18/7/15: 15
Only in /mnt/lustre/racer/14/6/18/7/15: 16
diff: /mnt/lustre/racer/14/6/18/7/15/18: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7/15: 3
diff: /mnt/lustre/racer/14/6/18/7/15/4: No such file or directory
diff: /mnt/lustre/racer/14/6/18/7/15/6: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/6: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/7: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/7: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/8: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/8: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/6/18/7/15/9: Too many levels of symbolic links
diff: /home/target/racer/14/6/18/7/15/9: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/6/18/7: 19
Only in /mnt/lustre/racer/14/6/18/7: 2
Only in /mnt/lustre/racer/14/6/18/7: 6
Only in /mnt/lustre/racer/14/6/18/7/7: 0
Only in /mnt/lustre/racer/14/6/18/7/7: 11
Only in /mnt/lustre/racer/14/6/18/7/7: 17
Only in /mnt/lustre/racer/14/6/18/7/7: 2
Only in /mnt/lustre/racer/14/6/18/7/7: 6
Only in /mnt/lustre/racer/14/6/18/7: 9
diff: /mnt/lustre/racer/14/6/18/8: No such file or directory
diff: /home/target/racer/14/6/18/8: No such file or directory
Only in /mnt/lustre/racer/14/6/18/9: 0
Only in /mnt/lustre/racer/14/6/18/9: 10
Only in /mnt/lustre/racer/14/6/18/9: 11
Only in /mnt/lustre/racer/14/6/18/9: 15
Only in /mnt/lustre/racer/14/6/18/9: 16
Only in /mnt/lustre/racer/14/6/18/9: 19
Only in /mnt/lustre/racer/14/6/18/9: 3
Only in /mnt/lustre/racer/14/6: 3
Only in /mnt/lustre/racer/14: 7
Only in /mnt/lustre/racer/14/8: 1
diff: /mnt/lustre/racer/14/8/10: Too many levels of symbolic links
diff: /home/target/racer/14/8/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/8/11/8: No such file or directory
diff: /home/target/racer/14/8/11/8: No such file or directory
Only in /mnt/lustre/racer/14/8/11: 9
Only in /mnt/lustre/racer/14/8: 13
Only in /mnt/lustre/racer/14/8: 14
diff: /mnt/lustre/racer/14/8/15: Too many levels of symbolic links
diff: /home/target/racer/14/8/15: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8: 16
Only in /mnt/lustre/racer/14/8: 17
Only in /mnt/lustre/racer/14/8: 18
Only in /mnt/lustre/racer/14/8: 19
diff: /mnt/lustre/racer/14/8/4/0: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/0: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 1
Only in /mnt/lustre/racer/14/8/4: 10
diff: /mnt/lustre/racer/14/8/4/11: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/11: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 12
diff: /mnt/lustre/racer/14/8/4/13: No such file or directory
Only in /mnt/lustre/racer/14/8/4: 15
Only in /mnt/lustre/racer/14/8/4: 16
diff: /mnt/lustre/racer/14/8/4/17: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/17: Too many levels of symbolic links
diff: /mnt/lustre/racer/14/8/4/18: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/18: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 19
Only in /mnt/lustre/racer/14/8/4: 2
Only in /mnt/lustre/racer/14/8/4: 3
diff: /mnt/lustre/racer/14/8/4/5: Too many levels of symbolic links
diff: /home/target/racer/14/8/4/5: Too many levels of symbolic links
Only in /mnt/lustre/racer/14/8/4: 6
Only in /mnt/lustre/racer/14/8/4: 8
Only in /mnt/lustre/racer/14/8/4: 9
Only in /mnt/lustre/racer/14/8: 6
Only in /mnt/lustre/racer/14/8: 7
Only in /mnt/lustre/racer/14: 9
diff: /home/target/racer/15: No such file or directory
Only in /mnt/lustre/racer: 16
Only in /mnt/lustre/racer/17: 0
Only in /mnt/lustre/racer/17: 1
Only in /mnt/lustre/racer/17: 10
Only in /mnt/lustre/racer/17: 13
Only in /mnt/lustre/racer/17: 16
Only in /mnt/lustre/racer/17: 17
diff: /mnt/lustre/racer/17/18: No such file or directory
Only in /mnt/lustre/racer/17: 19
diff: /mnt/lustre/racer/17/2: No such file or directory
diff: /home/target/racer/17/2: No such file or directory
Only in /home/target/racer/17: 4
Only in /mnt/lustre/racer/17: 5
Only in /mnt/lustre/racer/17: 7
Only in /mnt/lustre/racer/17: 8
Only in /mnt/lustre/racer/17: 9
Only in /mnt/lustre/racer/18: 1
Only in /mnt/lustre/racer/18: 12
Only in /mnt/lustre/racer/18: 16
Only in /home/target/racer/18: 2
diff: /mnt/lustre/racer/18/4: Too many levels of symbolic links
diff: /home/target/racer/18/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/18: 8
Only in /home/target/racer: 2
Only in /mnt/lustre/racer/3: 1
Only in /mnt/lustre/racer/3: 12
Only in /mnt/lustre/racer/3: 16
Only in /home/target/racer/3: 2
diff: /mnt/lustre/racer/3/4: Too many levels of symbolic links
diff: /home/target/racer/3/4: Too many levels of symbolic links
Only in /mnt/lustre/racer/3: 8
diff: /mnt/lustre/racer/4/10: Too many levels of symbolic links
diff: /home/target/racer/4/10: Too many levels of symbolic links
diff: /mnt/lustre/racer/4/7: No such file or directory
diff: /home/target/racer/4/7: No such file or directory
diff: /mnt/lustre/racer/5: Too many levels of symbolic links
diff: /mnt/lustre/racer/8: Too many levels of symbolic links
Only in /mnt/lustre/racer/9: 15
diff: /mnt/lustre/racer/9/2: Too many levels of symbolic links
diff: /home/target/racer/9/2: Too many levels of symbolic links
diff: /mnt/lustre/racer/9/6: Too many levels of symbolic links
diff: /home/target/racer/9/6: Too many levels of symbolic links
lustre-rsync-bug test_13: @@@@@@ FAIL: Failure in replication; differences found.
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:4343:error_noexit()
= /usr/lib64/lustre/tests/test-framework.sh:4374:error()
= lustre-rsync-bug.sh:144:check_diff()
= lustre-rsync-bug.sh:203:test_13()
= /usr/lib64/lustre/tests/test-framework.sh:4613:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:4648:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:4516:run_test()
= lustre-rsync-bug.sh:210:main()
Dumping lctl log to /home/test_logs/1423558610/lustre-rsync-bug.test_13.*.1423558690.log

The attached file is the test script.

Comments

[Joseph Gmitter (Inactive)]

Landed for 2.8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/14914/
Subject: LU-6229 utils: fix lustre_rsync bug of cascade move
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b9f3c954ef00c83ea34d0e3d7474f2b27eac29b8

[Niu Yawei (Inactive)]

Thank you for the patch, LiXi.

[Li Xi (Inactive)]

The script that can preproduce the problem easily.

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14914
Subject: LU-6229 utils: fix lustre_rsync bug of cascade move
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 055b39c94032bccbb6caf6f0308f08ae8dae64e3

[Li Xi (Inactive)]

I might be wrong, but the following might be incorrect:

* 1. creat
 *    If tfid is absent on the source-fs, ignore this operation
 *    If pfid is absent on the source-fs [or]
 *    if f2p(pfid) is not present on target-fs [or]
 *    if f2p(pfid)+name != f2p(tfid)
 *      creat .lustrerepl/tfid
 *      track [pfid,tfid,name]
 *    Else
 *      creat f2p[tfid]

In the case of 'if f2p(pfid)+name != f2p(tfid)', we need to create file tfid with path of "f2p(pfid)+name". I will make a patch and check whether this is the cause.

[wu libin (Inactive)]

yeah, single thread of this script will cause the problem.

Oleg, any suggestion about this problem?

[Li Xi (Inactive)]

Following is how lustre_rsync works (Copied from the codes):

* 1. creat
 *    If tfid is absent on the source-fs, ignore this operation
 *    If pfid is absent on the source-fs [or]
 *    if f2p(pfid) is not present on target-fs [or]
 *    if f2p(pfid)+name != f2p(tfid)
 *      creat .lustrerepl/tfid
 *      track [pfid,tfid,name]
 *    Else
 *      creat f2p[tfid]
 *
 * 2. remove
 *    If .lustrerepl/[tfid] is present on the target
 *      rm .lustrerepl/[tfid]
 *    Else if pfid is present on the source-fs,
 *      if f2p(pfid)+name is present,
 *        rm f2p(pfid)+name
 *
 * 3. move (spfid,sname) to (pfid,name)
 *    If pfid is present
 *      if spfid is also present, mv (spfid,sname) to (pfid,name)
 *      else mv .lustrerepl/[sfid] to (pfid,name)
 *    Else if pfid is not present,
 *      if spfid is present, mv (spfid,sname) .lustrerepl/[sfid]
 *      move out all its children in .lustrerepl.
 *      [pfid,tfid,name] tracked from (1) is used for this.

And following is how the tree is created and replayed

1. mkdir racer			racer
2. mkdir racer/11		racer/11
3. mkdir racer/13		racer/11 racer/13
4. mv race/13 racer/14	racer/11 race/14
5. mkdir racer/14/14	racer/11 race/14/14
6. mv racer/14 racer/11	racer/11/14/14


1. mkdir racer			racer
2. mkdir racer/11		racer/11
3. mkdir racer/13		racer/11 .lustrerpl/tfidp[13]
4. mv racer/13 racer/14	racer/11 racer/14
5. mkdir racer/14/14	racer/11 racer/14 .lustrerpl/tfidp[14]
6. mv racer/14 racer/11	racer/11/14 .lustrerpl/tfidp[14]

So it seems the step 4 of replay goes wrong already. Libin, are we able to reproduce with a single thread script?

[wu libin (Inactive)]

Yeah, i'm try to fix it myself, but i did not find the root cause, i'm glad that if someone here could help me.

[Oleg Drokin]

I see. thanks for these steps.
Is this something you plan to fix yourself or do you need some help here?