[LU-3573] lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.1, Lustre 2.7.0, Lustre 2.5.3
• Labels: prz zfs
• Environment: a patch pushed to autotest
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Keith Mannthey (Inactive)

Description

On ZFS an error was seen with lustre-rsync test 8 the logs are here:
https://maloo.whamcloud.com/test_sets/e3492890-e901-11e2-ae91-52540035b04c

I don't really know how to read lrsync_log.

The test error reports a very basic:

== lustre-rsync-test test 8: Replicate multiple file/directory moves == 16:00:59 (1373410859)
CMD: wtm-10vm7 lctl --device lustre-MDT0000 changelog_register -n
lustre-MDT0000: Registered changelog user cl13
CMD: wtm-10vm7 lctl get_param -n mdd.lustre-MDT0000.changelog_users
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /tmp/target
Statuslog: /tmp/lustre_rsync.log
Changelog registration: cl13
Starting changelog record: 0
Clear changelog after use: no
Errors: 0
lustre_rsync took 107 seconds
Changelog records consumed: 1881
Only in /tmp/target/d0.lustre-rsync-test/d8/d08/d083: a3
 lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4066:error_noexit()
  = /usr/lib64/lustre/tests/test-framework.sh:4093:error()
  .....

Out of the last 100 runs it reports 1 error so it could be related to the base patch or a rare error.

Issue Links

• is duplicated by: LU-5765 sanity test_123a test_123b: rm: no such file or directory
• is related to: LU-5924 conf-sanity test_32b: list verification failed
• is related to: LU-6101 sanity test_24A: Can not delete directories
• is related to: LU-5475 readdir missing a directory
• is related to: LU-3580 Panic in ptlrpc when rerunning lustre-rsync-test/8 without remount
• is related to: LU-5254 readdir missing a directory

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13357/
Subject: LU-3573 osd-zfs: Only advance zap cursor as needed
Project: fs/lustre-release
Branch: b2_5
Current Patch Set:
Commit: ce75d8f368a506993730da4224f4cfc18c2a0260

[Gerrit Updater]

Bob Glossman ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13357
Subject: LU-3573 osd-zfs: Only advance zap cursor as needed
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 78d6d3aeea75334fa7af3332c2835fcf573a8a7a

[Jodi Levi (Inactive)]

Patches landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12904/
Subject: LU-3573 osd-zfs: Only advance zap cursor as needed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0780fde4d415aa1f407ed0094f20f30e6f1b0d3a

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12887/
Subject: Revert "LU-3573 osd-zfs: Only advance zap cursor as needed"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 038d747f2ac5971875734701f5319047d6703844

[Nathaniel Clark]

When http://review.whamcloud.com/12887 is landed this bug will be open

[Nathaniel Clark]

This should be a fix for this bug without re-introducing LU-5924
http://review.whamcloud.com/12904

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12904
Subject: LU-3573 osd-zfs: Only advance zap cursor as needed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 90a5fa16f900ef24412bad534c848e99e091e44b

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12887
Subject: Revert "LU-3573 osd-zfs: Only advance zap cursor as needed"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5f6d6b371a18a8b164f0414a5dc7356f066fd2c4

[Jodi Levi (Inactive)]

Patch landed to Master. b2_5 patch ready to land and tracked outside of this ticket.

[Andreas Dilger]

Unfortunately, version 2 of the http://review.whamcloud.com/12582 patch failed review-zfs conf-sanity test_32b with a real problem that is now causing serious test failures on master. The fact that it failed on the unlanded patch, when no other tests were failing before that time makes it very likely that this patch caused the problem. I've opened LU-5924 for that issue, but we might consider reverting this patch if it cannot be addressed quickly.