[LU-3582] Runtests failed: old and new files are different: rc=11

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.5.0, Lustre 2.4.2
• Affects Version/s: Lustre 2.4.1, Lustre 2.5.0
• Labels: HB
• Severity: 3
• Assignee: Keith Mannthey (Inactive)
• Reporter: Keith Mannthey (Inactive)

Description

https://maloo.whamcloud.com/test_sets/e97cb600-eb22-11e2-847f-52540035b04c

From the set logs:

11:15:04:Files /bin/date and /mnt/lustre/runtest.22993//bin/date differ
11:15:04:Files /bin/rmdir and /mnt/lustre/runtest.22993//bin/rmdir differ
11:15:04:Files /bin/sed and /mnt/lustre/runtest.22993//bin/sed differ
11:15:04: runtests : @@@@@@ FAIL: old and new files are different: rc=11 
11:15:04:  Trace dump:
11:15:05:  = /usr/lib64/lustre/tests/test-framework.sh:4057:error_noexit()
11:15:05:  = /usr/lib64/lustre/tests/test-framework.sh:4084:error()
11:15:05:  = /usr/lib64/lustre/tests/test-framework.sh:4089:error_exit()
11:15:05:  = /usr/lib64/lustre/tests/runtests:84:main()
11:15:05:Dumping lctl log to /logdir/test_logs/2013-07-11/lustre-reviews-el6-x86_64--review-zfs--1_3_1__16600__-70253551900780-174227/runtests..*.1373652893.log
11:15:05:CMD: wtm-15vm3,wtm-15vm4,wtm-15vm5,wtm-15vm6.rosso.whamcloud.com /usr/sbin/lctl dk > /logdir/test_logs/2013-07-11/lustre-reviews-el6-x86_64--review-zfs--1_3_1__16600__-70253551900780-174227/runtests..debug_log.\$(hostname -s).1373652893.log;
11:15:05:         dmesg > /logdir/test_logs/2013-07-11/lustre-reviews-el6-x86_64--review-zfs--1_3_1__16600__-70253551900780-174227/runtests..dmesg.\$(hostname -s).1373652893.log
11:15:16:status        script            Total(sec) E(xcluded) S(low) 
11:15:16:------------------------------------------------------------------------------------
11:15:16:test-framework exiting on error
11:15:16:runtests returned 0
11:15:16:running: sanity-hsm 
11:15:16:run_suite sanity-hsm /usr/lib64/lustre/tests/sanity-hsm.sh

The test fails.

The auto test logging is pretty minimal for this test.

Issue Links

• is related to: LU-3895 conf-sanity test 47: umount -d -f /mnt/ost1
• is related to: LU-3126 conf-sanity test_41b: fld_server_lookup()) ASSERTION( fld->lsf_control_exp ) failed
• is related to: LU-3706 sanity-scrub test_13 timeout

Comments

[Gerrit Updater]

Minh Diep ([e-mail]) merged in patch https://review-stage.hpdd.intel.com/28009/
Subject: LU-3582 mdc: add cl_device to the MDC
Project: fs/lustre-release
Branch: dom
Current Patch Set:
Commit: f5510004de33ab768d016f6d26612b005680eb4d

[Gerrit Updater]

Minh Diep ([e-mail]) merged in patch https://review-stage.hpdd.intel.com/28009/
Subject: LU-3582 mdc: add cl_device to the MDC
Project: fs/lustre-release
Branch: dom
Current Patch Set:
Commit: 8adbb3aa6cdddd84f1a94ff40e76406a441e9336

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28009
Subject: LU-3582 mdc: add cl_device to the MDC
Project: fs/lustre-release
Branch: dom
Current Patch Set: 1
Commit: 8271c53f0ff7c68b6c175ffb7462e55c6904dbeb

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22997
Subject: LU-3582 mdc: add cl_device to the MDC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 642730ade7da65f86a8b57100c86e2241d81569a

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/58/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs
MDSCOUNT=1

runtests failed again:
https://maloo.whamcloud.com/test_sets/275948da-5603-11e3-8e94-52540035b04c

I ran the test with the same configuration for 6 times manually, all runs passed:
https://maloo.whamcloud.com/test_sessions/d184f670-567f-11e3-82a8-52540035b04c
https://maloo.whamcloud.com/test_sessions/cfcafbe0-567f-11e3-9304-52540035b04c
https://maloo.whamcloud.com/test_sessions/cdad4016-567f-11e3-9304-52540035b04c
https://maloo.whamcloud.com/test_sessions/cd3494b8-567f-11e3-9288-52540035b04c
https://maloo.whamcloud.com/test_sessions/cb6fd476-567f-11e3-9304-52540035b04c
https://maloo.whamcloud.com/test_sessions/cad680fa-567f-11e3-9288-52540035b04c

[Jian Yu]

Patch http://review.whamcloud.com/7014 needs to be cherry-picked to Lustre b2_4 branch so that logs can be uploaded to Maloo next time the test fails.

Patch for Lustre b2_4 branch is in http://review.whamcloud.com/8384.

[Jodi Levi (Inactive)]

Patches landed to Master and issue does not seem to be occurring any longer.

[Jodi Levi (Inactive)]

Reducing priority for a while longer to see if this occurs again and then will close.

[Keith Mannthey (Inactive)]

Sorry for the late response.

We don't seem to be hitting this issue right now. I looked at Maloo on friday and it had been almost 2 weeks since we hand seen issues. It is unclear what has happened but we are definitely not hitting this issue like we were.

[Jodi Levi (Inactive)]

Is this still happening regularly?