[LU-6804] sanity test_27p: @@@@@@ FAIL: checkstat failed

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Ashish Purkar
• Assignee: WC Triage
• Resolution: Cannot Reproduce
• Labels: patch
• Epic/Theme: test
• Severity: 3

Description

In sanity test, test_27p is failing with "checkstat failed" error message.

== sanity test 27p: append to a truncated file with some full OSTs ===== 11:22:41 (1393068161)
fail_loc=0
/mnt/lustre/d27p.sanity/f27p.sanity has size 80000000 OK
OSTIDX=0 MDSIDX=1
pdsh@mft63: mft61: ssh exited with exit code 3
pdsh@mft63: mft61: ssh exited with exit code 3
osc.lustre-OST0000-osc-MDT0000.prealloc_last_id=353
osc.lustre-OST0000-osc-MDT0000.prealloc_last_seq=0x100000000
osc.lustre-OST0000-osc-MDT0000.prealloc_next_id=324
osc.lustre-OST0000-osc-MDT0000.prealloc_next_seq=0x100000000
osc.lustre-OST0000-osc-MDT0000.prealloc_reserved=0
osc.lustre-OST0000-osc-MDT0000.prealloc_status=0
osc.lustre-OST0001-osc-MDT0000.prealloc_last_id=65
osc.lustre-OST0001-osc-MDT0000.prealloc_last_seq=0x100010000
osc.lustre-OST0001-osc-MDT0000.prealloc_next_id=35
osc.lustre-OST0001-osc-MDT0000.prealloc_next_seq=0x100010000
osc.lustre-OST0001-osc-MDT0000.prealloc_reserved=0
osc.lustre-OST0001-osc-MDT0000.prealloc_status=0
fail_val=0
fail_loc=0x215
Creating to objid 353 on ost lustre-OST0000...
total: 31 creates in 0.04 seconds: 760.81 creates/second
osc.lustre-OST0000-osc-MDT0000.prealloc_last_id=385
osc.lustre-OST0000-osc-MDT0000.prealloc_last_seq=0x100000000
osc.lustre-OST0000-osc-MDT0000.prealloc_next_id=355
osc.lustre-OST0000-osc-MDT0000.prealloc_next_seq=0x100000000
osc.lustre-OST0000-osc-MDT0000.prealloc_reserved=0
osc.lustre-OST0000-osc-MDT0000.prealloc_status=0
osc.lustre-OST0001-osc-MDT0000.prealloc_last_id=65
osc.lustre-OST0001-osc-MDT0000.prealloc_last_seq=0x100010000
osc.lustre-OST0001-osc-MDT0000.prealloc_next_id=35
osc.lustre-OST0001-osc-MDT0000.prealloc_next_seq=0x100010000
osc.lustre-OST0001-osc-MDT0000.prealloc_reserved=0
osc.lustre-OST0001-osc-MDT0000.prealloc_status=0
fail_loc=0x80000215
/mnt/lustre/d27p.sanity/f27p.sanity has size 80000000, not 80000004
sanity test_27p: @@@@@@ FAIL: checkstat failed

Comments

[Gerrit Updater]

Ashish Purkar ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15520
Subject: LU-6804 tests: Make sure to flush cache
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: abfd03d6cfcfe747e5491d9746dabc19841eb3c1

[Andreas Dilger]

I don't think this patch is the correct approach at all. At best it is hiding a serious bug, but it isn't actually fixing the apparent data corruption. There should not be any reason to flush the cache in order to keep the visible file size consistent on any client node. The DLM should handle this transparently to userspace.

[Andreas Dilger]

Is there something peculiar about your test configuration? We are not seeing any test failures for test_27p in recent history. It may also be caused by patches in your local tree.

[Ashish Purkar]

This problem was reproducible intermittently on local tree having few different patches and test configuration was quartet regular setup and with dne as well it was reproducible.

I tried to reproduce manually on master like below steps but could not reproduce it.
[root@maximus lustre-wc-rel]# git describe
2.7.56-53-gfe60e01

[root@maximus lustre]# touch /mnt/lustre/test
[root@maximus lustre]# stat /mnt/lustre/test
File: `/mnt/lustre/test'
Size: 0 Blocks: 0 IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1
Access: (0644/rw-rr-) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-24 17:29:19.000000000 +0900
Modify: 2015-07-24 17:29:19.000000000 +0900
Change: 2015-07-24 17:29:19.000000000 +0900

[root@maximus lustre]# /root/code/lustre-wc-rel/lustre/tests/truncate /mnt/lustre/test
Usage: truncate PATH LENGTH
[root@maximus lustre]# /root/code/lustre-wc-rel/lustre/tests/truncate /mnt/lustre/test 80000000
[root@maximus lustre]# stat /mnt/lustre/test
File: `/mnt/lustre/test'
Size: 80000000 Blocks: 0 IO Block: 4194304 regular file
Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1
Access: (0644/rw-rr-) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-24 17:29:19.000000000 +0900
Modify: 2015-07-24 17:30:25.000000000 +0900
Change: 2015-07-24 17:30:25.000000000 +0900
[root@maximus lustre]# /root/code/lustre-wc-rel/lustre/tests/checkstat -s 80000
000 /mnt/lustre/test
[root@maximus lustre]# echo $?
0

[root@maximus lustre]# echo foo >> /mnt/lustre/test
[root@maximus lustre]# stat /mnt/lustre/test
File: `/mnt/lustre/test'
Size: 80000004 Blocks: 1 IO Block: 4194304 regular file
Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1
Access: (0644/rw-rr-) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-24 17:29:19.000000000 +0900
Modify: 2015-07-24 17:32:12.000000000 +0900
Change: 2015-07-24 17:32:12.000000000 +0900
[root@maximus lustre]# /root/code/lustre-wc-rel/lustre/tests/checkstat -s 80000004 /mnt/lustre/test
[root@maximus lustre]# echo $?
0