Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.8.0
    • None
    • 3
    • 17439

    Description

      As i describe in LU-6166, the lustre_rsync has some problems, here is test result with racer, the result is:
      Lustre filesystem: lustre
      MDT device: lustre-MDT0000
      Source: /mnt/lustre
      Target: /home/target
      Statuslog: /home/lustre_rsync.log
      Changelog registration: cl1
      Starting changelog record: 0
      Clear changelog after use: no
      Errors: 1057
      lustre_rsync took 36 seconds
      Changelog records consumed: 6774
      Only in /mnt/lustre: .lustre
      Only in /home/target: .lustrerepl
      Only in /mnt/lustre/racer: 0
      diff: /mnt/lustre/racer/1: Too many levels of symbolic links
      Only in /home/target/racer: 10
      diff: /mnt/lustre/racer/11/12: Too many levels of symbolic links
      diff: /home/target/racer/11/12: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/13: Too many levels of symbolic links
      diff: /home/target/racer/11/13: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/1: Too many levels of symbolic links
      diff: /home/target/racer/11/16/1: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/10/0: No such file or directory
      diff: /home/target/racer/11/16/10/0: No such file or directory
      Only in /mnt/lustre/racer/11/16/10: 13
      diff: /mnt/lustre/racer/11/16/10/18: Not a directory
      diff: /home/target/racer/11/16/10/18: Not a directory
      diff: /mnt/lustre/racer/11/16/10/6: Too many levels of symbolic links
      diff: /home/target/racer/11/16/10/6: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16: 12
      diff: /mnt/lustre/racer/11/16/13: Too many levels of symbolic links
      diff: /home/target/racer/11/16/13: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16: 15
      diff: /mnt/lustre/racer/11/16/17: Too many levels of symbolic links
      diff: /home/target/racer/11/16/17: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16: 18
      Only in /mnt/lustre/racer/11/16: 19
      diff: /mnt/lustre/racer/11/16/2: Too many levels of symbolic links
      diff: /home/target/racer/11/16/2: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/3: No such file or directory
      diff: /home/target/racer/11/16/3: No such file or directory
      diff: /mnt/lustre/racer/11/16/4: Too many levels of symbolic links
      diff: /home/target/racer/11/16/4: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16: 5
      Only in /mnt/lustre/racer/11/16: 6
      diff: /mnt/lustre/racer/11/16/7/12: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/12: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/7/19: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/19: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16/7/8: 0
      diff: /mnt/lustre/racer/11/16/7/8/10: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/8/10: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/7/8/16: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/8/16: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16/7/8: 2
      Only in /mnt/lustre/racer/11/16/7/8/4: 11
      Only in /mnt/lustre/racer/11/16/7/8/4: 19
      diff: /mnt/lustre/racer/11/16/7/8/4/6: No such file or directory
      diff: /home/target/racer/11/16/7/8/4/6: No such file or directory
      diff: /mnt/lustre/racer/11/16/7/8/5: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/8/5: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/7/8/7: Too many levels of symbolic links
      diff: /home/target/racer/11/16/7/8/7: Too many levels of symbolic links
      diff: /mnt/lustre/racer/11/16/8: Too many levels of symbolic links
      diff: /home/target/racer/11/16/8: Too many levels of symbolic links
      Only in /mnt/lustre/racer/11/16: 9
      diff: /mnt/lustre/racer/11/19: Too many levels of symbolic links
      diff: /home/target/racer/11/19: Too many levels of symbolic links
      diff: /home/target/racer/12: No such file or directory
      Only in /mnt/lustre/racer/14: 0
      Only in /mnt/lustre/racer/14: 1
      Only in /mnt/lustre/racer/14: 10
      Only in /mnt/lustre/racer/14: 11
      Only in /mnt/lustre/racer/14/12: 15
      diff: /mnt/lustre/racer/14/12/3: Too many levels of symbolic links
      diff: /home/target/racer/14/12/3: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/12/5: Too many levels of symbolic links
      diff: /home/target/racer/14/12/5: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/12: 6
      diff: /mnt/lustre/racer/14/12/8: Too many levels of symbolic links
      diff: /home/target/racer/14/12/8: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14: 14
      Only in /mnt/lustre/racer/14/16: 0
      Only in /mnt/lustre/racer/14/16: 1
      diff: /mnt/lustre/racer/14/16/6: Too many levels of symbolic links
      diff: /home/target/racer/14/16/6: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/17: Too many levels of symbolic links
      diff: /home/target/racer/14/17: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14: 18
      Only in /mnt/lustre/racer/14/3: 1
      Only in /mnt/lustre/racer/14/3: 10
      diff: /mnt/lustre/racer/14/3/11: Too many levels of symbolic links
      diff: /home/target/racer/14/3/11: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/3: 12
      Only in /mnt/lustre/racer/14/3: 13
      Only in /mnt/lustre/racer/14/3: 14
      Only in /mnt/lustre/racer/14/3: 16
      Only in /mnt/lustre/racer/14/3: 17
      diff: /mnt/lustre/racer/14/3/19: Too many levels of symbolic links
      diff: /home/target/racer/14/3/19: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/3: 2
      Only in /mnt/lustre/racer/14/3: 3
      Only in /mnt/lustre/racer/14/3: 4
      Only in /mnt/lustre/racer/14/3: 7
      Only in /mnt/lustre/racer/14/3: 8
      Only in /mnt/lustre/racer/14/3: 9
      Only in /mnt/lustre/racer/14: 4
      Only in /mnt/lustre/racer/14/5: 0
      Only in /mnt/lustre/racer/14/5: 1
      Only in /mnt/lustre/racer/14/5: 11
      Only in /mnt/lustre/racer/14/5: 12
      diff: /mnt/lustre/racer/14/5/13: Too many levels of symbolic links
      diff: /home/target/racer/14/5/13: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/5: 14
      File /mnt/lustre/racer/14/5/16 is a directory while file /home/target/racer/14/5/16 is a regular empty file
      diff: /mnt/lustre/racer/14/5/3: Too many levels of symbolic links
      diff: /home/target/racer/14/5/3: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/5: 6
      Only in /mnt/lustre/racer/14/5: 7
      Only in /mnt/lustre/racer/14/5: 9
      Only in /mnt/lustre/racer/14/6: 1
      Only in /mnt/lustre/racer/14/6: 12
      Only in /mnt/lustre/racer/14/6: 14
      Only in /mnt/lustre/racer/14/6: 16
      diff: /mnt/lustre/racer/14/6/18/0: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/0: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/10: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/10: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18: 11
      diff: /mnt/lustre/racer/14/6/18/12: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/12: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/13/11: No such file or directory
      diff: /home/target/racer/14/6/18/13/11: No such file or directory
      diff: /mnt/lustre/racer/14/6/18/13/12: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/13/12: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/13/15: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/13/15: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/13: 17
      diff: /mnt/lustre/racer/14/6/18/13/18: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/13/18: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/13/4: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/13/4: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/13: 5
      diff: /mnt/lustre/racer/14/6/18/13/7: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/13/7: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/13: 8
      Only in /mnt/lustre/racer/14/6/18/13: 9
      Only in /mnt/lustre/racer/14/6/18: 16
      diff: /mnt/lustre/racer/14/6/18/18: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/18: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18: 19
      Only in /mnt/lustre/racer/14/6/18: 2
      Only in /mnt/lustre/racer/14/6/18: 3
      diff: /mnt/lustre/racer/14/6/18/4: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/4: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18: 5
      diff: /mnt/lustre/racer/14/6/18/7/10: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/10: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/7: 12
      Only in /mnt/lustre/racer/14/6/18/7: 14
      Only in /mnt/lustre/racer/14/6/18/7/15: 0
      Only in /mnt/lustre/racer/14/6/18/7/15: 1
      diff: /mnt/lustre/racer/14/6/18/7/15/10: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/10: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/7/15/11: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/11: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/7/15/12: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/12: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 14
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 19
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 2
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 4
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 5
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 6
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 7
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 8
      Only in /mnt/lustre/racer/14/6/18/7/15/13: 9
      Only in /mnt/lustre/racer/14/6/18/7/15: 14
      Only in /mnt/lustre/racer/14/6/18/7/15: 15
      Only in /mnt/lustre/racer/14/6/18/7/15: 16
      diff: /mnt/lustre/racer/14/6/18/7/15/18: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/18: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/7/15: 3
      diff: /mnt/lustre/racer/14/6/18/7/15/4: No such file or directory
      diff: /mnt/lustre/racer/14/6/18/7/15/6: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/6: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/7/15/7: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/7: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/7/15/8: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/8: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/6/18/7/15/9: Too many levels of symbolic links
      diff: /home/target/racer/14/6/18/7/15/9: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/6/18/7: 19
      Only in /mnt/lustre/racer/14/6/18/7: 2
      Only in /mnt/lustre/racer/14/6/18/7: 6
      Only in /mnt/lustre/racer/14/6/18/7/7: 0
      Only in /mnt/lustre/racer/14/6/18/7/7: 11
      Only in /mnt/lustre/racer/14/6/18/7/7: 17
      Only in /mnt/lustre/racer/14/6/18/7/7: 2
      Only in /mnt/lustre/racer/14/6/18/7/7: 6
      Only in /mnt/lustre/racer/14/6/18/7: 9
      diff: /mnt/lustre/racer/14/6/18/8: No such file or directory
      diff: /home/target/racer/14/6/18/8: No such file or directory
      Only in /mnt/lustre/racer/14/6/18/9: 0
      Only in /mnt/lustre/racer/14/6/18/9: 10
      Only in /mnt/lustre/racer/14/6/18/9: 11
      Only in /mnt/lustre/racer/14/6/18/9: 15
      Only in /mnt/lustre/racer/14/6/18/9: 16
      Only in /mnt/lustre/racer/14/6/18/9: 19
      Only in /mnt/lustre/racer/14/6/18/9: 3
      Only in /mnt/lustre/racer/14/6: 3
      Only in /mnt/lustre/racer/14: 7
      Only in /mnt/lustre/racer/14/8: 1
      diff: /mnt/lustre/racer/14/8/10: Too many levels of symbolic links
      diff: /home/target/racer/14/8/10: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/8/11/8: No such file or directory
      diff: /home/target/racer/14/8/11/8: No such file or directory
      Only in /mnt/lustre/racer/14/8/11: 9
      Only in /mnt/lustre/racer/14/8: 13
      Only in /mnt/lustre/racer/14/8: 14
      diff: /mnt/lustre/racer/14/8/15: Too many levels of symbolic links
      diff: /home/target/racer/14/8/15: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/8: 16
      Only in /mnt/lustre/racer/14/8: 17
      Only in /mnt/lustre/racer/14/8: 18
      Only in /mnt/lustre/racer/14/8: 19
      diff: /mnt/lustre/racer/14/8/4/0: Too many levels of symbolic links
      diff: /home/target/racer/14/8/4/0: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/8/4: 1
      Only in /mnt/lustre/racer/14/8/4: 10
      diff: /mnt/lustre/racer/14/8/4/11: Too many levels of symbolic links
      diff: /home/target/racer/14/8/4/11: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/8/4: 12
      diff: /mnt/lustre/racer/14/8/4/13: No such file or directory
      Only in /mnt/lustre/racer/14/8/4: 15
      Only in /mnt/lustre/racer/14/8/4: 16
      diff: /mnt/lustre/racer/14/8/4/17: Too many levels of symbolic links
      diff: /home/target/racer/14/8/4/17: Too many levels of symbolic links
      diff: /mnt/lustre/racer/14/8/4/18: Too many levels of symbolic links
      diff: /home/target/racer/14/8/4/18: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/8/4: 19
      Only in /mnt/lustre/racer/14/8/4: 2
      Only in /mnt/lustre/racer/14/8/4: 3
      diff: /mnt/lustre/racer/14/8/4/5: Too many levels of symbolic links
      diff: /home/target/racer/14/8/4/5: Too many levels of symbolic links
      Only in /mnt/lustre/racer/14/8/4: 6
      Only in /mnt/lustre/racer/14/8/4: 8
      Only in /mnt/lustre/racer/14/8/4: 9
      Only in /mnt/lustre/racer/14/8: 6
      Only in /mnt/lustre/racer/14/8: 7
      Only in /mnt/lustre/racer/14: 9
      diff: /home/target/racer/15: No such file or directory
      Only in /mnt/lustre/racer: 16
      Only in /mnt/lustre/racer/17: 0
      Only in /mnt/lustre/racer/17: 1
      Only in /mnt/lustre/racer/17: 10
      Only in /mnt/lustre/racer/17: 13
      Only in /mnt/lustre/racer/17: 16
      Only in /mnt/lustre/racer/17: 17
      diff: /mnt/lustre/racer/17/18: No such file or directory
      Only in /mnt/lustre/racer/17: 19
      diff: /mnt/lustre/racer/17/2: No such file or directory
      diff: /home/target/racer/17/2: No such file or directory
      Only in /home/target/racer/17: 4
      Only in /mnt/lustre/racer/17: 5
      Only in /mnt/lustre/racer/17: 7
      Only in /mnt/lustre/racer/17: 8
      Only in /mnt/lustre/racer/17: 9
      Only in /mnt/lustre/racer/18: 1
      Only in /mnt/lustre/racer/18: 12
      Only in /mnt/lustre/racer/18: 16
      Only in /home/target/racer/18: 2
      diff: /mnt/lustre/racer/18/4: Too many levels of symbolic links
      diff: /home/target/racer/18/4: Too many levels of symbolic links
      Only in /mnt/lustre/racer/18: 8
      Only in /home/target/racer: 2
      Only in /mnt/lustre/racer/3: 1
      Only in /mnt/lustre/racer/3: 12
      Only in /mnt/lustre/racer/3: 16
      Only in /home/target/racer/3: 2
      diff: /mnt/lustre/racer/3/4: Too many levels of symbolic links
      diff: /home/target/racer/3/4: Too many levels of symbolic links
      Only in /mnt/lustre/racer/3: 8
      diff: /mnt/lustre/racer/4/10: Too many levels of symbolic links
      diff: /home/target/racer/4/10: Too many levels of symbolic links
      diff: /mnt/lustre/racer/4/7: No such file or directory
      diff: /home/target/racer/4/7: No such file or directory
      diff: /mnt/lustre/racer/5: Too many levels of symbolic links
      diff: /mnt/lustre/racer/8: Too many levels of symbolic links
      Only in /mnt/lustre/racer/9: 15
      diff: /mnt/lustre/racer/9/2: Too many levels of symbolic links
      diff: /home/target/racer/9/2: Too many levels of symbolic links
      diff: /mnt/lustre/racer/9/6: Too many levels of symbolic links
      diff: /home/target/racer/9/6: Too many levels of symbolic links
      lustre-rsync-bug test_13: @@@@@@ FAIL: Failure in replication; differences found.
      Trace dump:
      = /usr/lib64/lustre/tests/test-framework.sh:4343:error_noexit()
      = /usr/lib64/lustre/tests/test-framework.sh:4374:error()
      = lustre-rsync-bug.sh:144:check_diff()
      = lustre-rsync-bug.sh:203:test_13()
      = /usr/lib64/lustre/tests/test-framework.sh:4613:run_one()
      = /usr/lib64/lustre/tests/test-framework.sh:4648:run_one_logged()
      = /usr/lib64/lustre/tests/test-framework.sh:4516:run_test()
      = lustre-rsync-bug.sh:210:main()
      Dumping lctl log to /home/test_logs/1423558610/lustre-rsync-bug.test_13.*.1423558690.log

      The attached file is the test script.

      Attachments

        Activity

          [LU-6229] test racer with lustre_rsync
          jgmitter Joseph Gmitter (Inactive) made changes -
          Fix Version/s New: Lustre 2.8.0 [ 11113 ]
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          Landed for 2.8

          jgmitter Joseph Gmitter (Inactive) added a comment - Landed for 2.8

          Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/14914/
          Subject: LU-6229 utils: fix lustre_rsync bug of cascade move
          Project: fs/lustre-release
          Branch: master
          Current Patch Set:
          Commit: b9f3c954ef00c83ea34d0e3d7474f2b27eac29b8

          gerrit Gerrit Updater added a comment - Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/14914/ Subject: LU-6229 utils: fix lustre_rsync bug of cascade move Project: fs/lustre-release Branch: master Current Patch Set: Commit: b9f3c954ef00c83ea34d0e3d7474f2b27eac29b8

          Thank you for the patch, LiXi.

          niu Niu Yawei (Inactive) added a comment - Thank you for the patch, LiXi.
          jfc John Fuchs-Chesney (Inactive) made changes -
          Labels New: patch
          lixi Li Xi (Inactive) made changes -
          Attachment New: test.sh [ 17921 ]

          The script that can preproduce the problem easily.

          lixi Li Xi (Inactive) added a comment - The script that can preproduce the problem easily.

          Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/14914
          Subject: LU-6229 utils: fix lustre_rsync bug of cascade move
          Project: fs/lustre-release
          Branch: master
          Current Patch Set: 1
          Commit: 055b39c94032bccbb6caf6f0308f08ae8dae64e3

          gerrit Gerrit Updater added a comment - Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/14914 Subject: LU-6229 utils: fix lustre_rsync bug of cascade move Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 055b39c94032bccbb6caf6f0308f08ae8dae64e3
          lixi Li Xi (Inactive) added a comment - - edited

          I might be wrong, but the following might be incorrect:

          * 1. creat
           *    If tfid is absent on the source-fs, ignore this operation
           *    If pfid is absent on the source-fs [or]
           *    if f2p(pfid) is not present on target-fs [or]
           *    if f2p(pfid)+name != f2p(tfid)
           *      creat .lustrerepl/tfid
           *      track [pfid,tfid,name]
           *    Else
           *      creat f2p[tfid]
          

          In the case of 'if f2p(pfid)+name != f2p(tfid)', we need to create file tfid with path of "f2p(pfid)+name". I will make a patch and check whether this is the cause.

          lixi Li Xi (Inactive) added a comment - - edited I might be wrong, but the following might be incorrect: * 1. creat * If tfid is absent on the source-fs, ignore this operation * If pfid is absent on the source-fs [or] * if f2p(pfid) is not present on target-fs [or] * if f2p(pfid)+name != f2p(tfid) * creat .lustrerepl/tfid * track [pfid,tfid,name] * Else * creat f2p[tfid] In the case of 'if f2p(pfid)+name != f2p(tfid)', we need to create file tfid with path of "f2p(pfid)+name". I will make a patch and check whether this is the cause.

          yeah, single thread of this script will cause the problem.

          Oleg, any suggestion about this problem?

          gnlwlb wu libin (Inactive) added a comment - yeah, single thread of this script will cause the problem. Oleg, any suggestion about this problem?

          People

            niu Niu Yawei (Inactive)
            gnlwlb wu libin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: