[LU-16489] lustre-rsync-test test_2c: Failure in replication; differences found

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Timothy Day <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/ed7b15d4-da4c-45e5-941d-171b43889c4c

test_2c failed with the following error:

Failure in replication; differences found.

Output:

Errors: 0
lustre_rsync took 1 seconds
Changelog records consumed: 173
Running lustre_rsync
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /tmp/target
Target: /tmp/target2
Statuslog: /tmp/lustre_rsync.log
Changelog registration: cl4
Starting changelog record: 7634
Clear changelog after use: no
Errors: 0
lustre_rsync took 1 seconds
Changelog records consumed: 2
Only in /tmp/target/d2c.lustre-rsync-test/clients/client0/~dmtmp/WORDPRO: BENCHS1.LWP
Only in /tmp/target/d2c.lustre-rsync-test/clients/client0/~dmtmp/WORDPRO: NEWS1_1A.LWP
lustre-rsync-test test_2c: @@@@@@ FAIL: Failure in replication; differences found. 
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:6549:error()
= /usr/lib64/lustre/tests/lustre-rsync-test.sh:113:check_diff()
= /usr/lib64/lustre/tests/lustre-rsync-test.sh:370:test_2c()
= /usr/lib64/lustre/tests/test-framework.sh:6887:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:6937:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:6773:run_test()
= /usr/lib64/lustre/tests/lustre-rsync-test.sh:377:main()
Dumping lctl log to /autotest/autotest-1/2023-01-17/lustre-reviews_review-dne-zfs-part-5_91682_16_90396e57-368d-4a37-bcb7-90f078a84f20//lustre-rsync-test.test_2c.*.1673986148.log
CMD: onyx-110vm6,onyx-110vm7,onyx-49vm6.onyx.whamcloud.com,onyx-49vm7,onyx-82vm6 /usr/sbin/lctl dk > /autotest/autotest-1/2023-01-17/lustre-reviews_review-dne-zfs-part-5_91682_16_90396e57-368d-4a37-bcb7-90f078a84f20//lustre-rsync-test.test_2c.debug_log.\$(hostname -s).1673986148.log;
dmesg > /autotest/autotest-1/2023-01-17/lustre-reviews_review-dne-zfs-part-5_91682_16_90396e57-368d-4a37-bcb7-90f078a84f20//lustre-rsync-test.test_2c.dmesg.\$(hostname -s).1673986148.log
CMD: onyx-110vm7 /usr/sbin/lctl get_param -n mdd.lustre-MDT0003.changelog_users
lustre-MDT0003: clear the changelog for cl4 of all records
CMD: onyx-110vm7 /usr/sbin/lctl --device lustre-MDT0003 changelog_deregister cl4
lustre-MDT0003: Deregistered changelog user #4
CMD: onyx-110vm7 /usr/sbin/lctl set_param mdd.lustre-MDT0003.changelog_mask='MARK ' -n
CMD: onyx-110vm6 /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.changelog_users
lustre-MDT0002: clear the changelog for cl4 of all records
CMD: onyx-110vm6 /usr/sbin/lctl --device lustre-MDT0002 changelog_deregister cl4
lustre-MDT0002: Deregistered changelog user #4
CMD: onyx-110vm6 /usr/sbin/lctl set_param mdd.lustre-MDT0002.changelog_mask='MARK ' -n
CMD: onyx-110vm7 /usr/sbin/lctl get_param -n mdd.lustre-MDT0001.changelog_users
lustre-MDT0001: clear the changelog for cl4 of all records
CMD: onyx-110vm7 /usr/sbin/lctl --device lustre-MDT0001 changelog_deregister cl4
lustre-MDT0001: Deregistered changelog user #4
CMD: onyx-110vm7 /usr/sbin/lctl set_param mdd.lustre-MDT0001.changelog_mask='MARK ' -n
CMD: onyx-110vm6 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users
lustre-MDT0000: clear the changelog for cl4 of all records
CMD: onyx-110vm6 /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl4
lustre-MDT0000: Deregistered changelog user #4
CMD: onyx-110vm6 /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask='MARK ' -n

Seems similar to https://jira.whamcloud.com/browse/LU-10054?jql=text%20~%20%22Failure%20in%20replication%3B%20differences%20found.%22 or https://jira.whamcloud.com/browse/LU-4256?jql=text%20~%20%22Failure%20in%20replication%3B%20differences%20found.%22.

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
lustre-rsync-test test_2c - Failure in replication; differences found.