[LU-2866] Interop 2.1.4<->2.4 failure on test suite lustre-rsync-test test_1: Failure in replication; differences found

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Affects Version/s: Lustre 2.4.0
• Labels: MB
• Environment: client: 2.1.4 RHEL6; server: lustre-master build #
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/2277b80e-7d52-11e2-a108-52540035b04c.

The sub-test test_1 failed with the following error:

Failure in replication; differences found.

client-1 test log shows:

== lustre-rsync-test test 1: Simple Replication == 04:47:54 (1361537274)
CMD: client-24vm3 lctl --device lustre-MDT0000 changelog_register -n
lustre-MDT0000: Registered changelog user cl1
CMD: client-24vm3 lctl get_param -n mdd.lustre-MDT0000.changelog_users
Replication #1
Replication of operation failed(-21): 15 UNLNK (6) [0x200000400:0x13:0x0] [0x200000400:0xd:0x0] file4
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /tmp/target
Target: /tmp/target2
Statuslog: /tmp/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 0
Errors: 1
lustre_rsync took 1 seconds
Changelog records consumed: 18
Replication #2
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/lustre
Target: /tmp/target
Target: /tmp/target2
Statuslog: /tmp/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 18
Errors: 0
lustre_rsync took 0 seconds
Changelog records consumed: 4
Only in /tmp/target/d0.lustre-rsync-test/d1/d1: file4
Only in /mnt/lustre/d0.lustre-rsync-test/d1: d2
 lustre-rsync-test test_1: @@@@@@ FAIL: Failure in replication; differences found. 
Dumping lctl log to /logdir/test_logs/2013-02-21/lustre-master-el6-x86_64-vs-lustre-b2_1-el6-x86_64--full--2_5_1__1269__-70264313904840-171517/lustre-rsync-

client console shows:

04:48:00:Lustre: DEBUG MARKER: == lustre-rsync-test test 1: Simple Replication == 04:47:54 (1361537274)
04:48:00:LustreError: 23091:0:(lmv_obd.c:855:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd c0086696: err = -2
04:48:00:LustreError: 23091:0:(lmv_obd.c:855:lmv_iocontrol()) error: iocontrol MDC lustre-MDT0000_UUID on MDTidx 0 cmd c0086696: err = -2
04:48:00:Lustre: DEBUG MARKER: /usr/sbin/lctl mark  lustre-rsync-test test_1: @@@@@@ FAIL: Failure in replication\; differences found. 
04:48:00:Lustre: DEBUG MARKER: lustre-rsync-test test_1: @@@@@@ FAIL: Failure in replication; differences found.

Issue Links

• duplicates: LU-1331 changelogs: RNMTO record not always after RNMFRM
• is related to: LU-1237 2.1.1<->2.2 interop Test failure on test suite lustre-rsync-test, subtest test_2c

Comments

[Peter Jones]

ok so I am closing this as a duplicate of LU-1331 and it will be fixed once b2_1 interop switches to 2.1.5

[Sarah Liu]

verified with the latest b2_1 build #191 and it passed:

https://maloo.whamcloud.com/test_sessions/f2de5456-9244-11e2-b7d5-52540035b04c

[Yang Sheng]

I think this issue should cause by 2.1.4-rc2 hasn't included http://review.whamcloud.com/#change,3835

So rename changelog is different between 2.1 & 2.4. Just include this patch else we have to stop the rsync in interop situation.

[Oleg Drokin]

So, I see LU-1526 is actually part of 2.1.4, then how will it be fixed in 2.1.5, by what patch?

[Sarah Liu]

Yang Sheng, I think this is the ticket LU-1526 and this issue should be fixed in 2.1.5

[Yang Sheng]

Ah, Obviously it isn't standarad operation, has a ticket for mkfs.lustre issue?

[Sarah Liu]

Hi Yang Sheng,

I can reproduce it on my test nodes, I just copy the test-framework.sh from server(2.4) to client(2.1.4), otherwise mkfs.lustre cannot work:
https://maloo.whamcloud.com/test_sets/403f19da-8ec4-11e2-92ff-52540035b04c

[Yang Sheng]

Hi, Sarah, have any special while the test running? I cannot reproduce it anyway.

[Yang Sheng]

I am still trying to reproduce it.

[Sarah Liu]

another instance: https://maloo.whamcloud.com/test_sets/bf2638e6-8906-11e2-b643-52540035b04c