[LU-12232] replay-ost-single test 6 fails with ''space grew after dd: before:13442048 after_dd:13442048'' Created: 26/Apr/19 Updated: 12/Oct/20 Resolved: 12/Oct/20 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.13.0, Lustre 2.12.1 |
| Fix Version/s: | Lustre 2.14.0 |
| Type: | Bug | Priority: | Minor |
| Reporter: | James Nunez (Inactive) | Assignee: | Hongchao Zhang |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | zfs | ||
| Issue Links: |
|
||||||||
| Severity: | 3 | ||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
replay-ost-single test 6 fails with ''space grew after dd: before:X after_dd:Y” for some values of X and Y. Looking at the suite_log for a recent failure, logs at https://testing.whamcloud.com/test_sets/d53ee48a-665d-11e9-8bb1-52540065bddc , we see CMD: trevis-45vm9 lctl set_param fail_loc=0x80000119 fail_loc=0x80000119 before: 13442048 after_dd: 13442048 took 20 seconds replay-ost-single test_6: @@@@@@ FAIL: space grew after dd: before:13442048 after_dd:13442048 Some of the failure have the before and after values the same and some failures have different values for before and after. There are no errors in any of the node console logs. This failure looks like There are several examples of this failure, but here are just a couple of additional links to logs |
| Comments |
| Comment by Peter Jones [ 29/Apr/19 ] |
|
Hongchao Could you please investigate? Peter |
| Comment by Patrick Farrell (Inactive) [ 29/Apr/19 ] |
|
Here's the failure check: log "before: $before after_dd: $after_dd took $i seconds"
(( $before > $after_dd )) ||
error "space grew after dd: before:$before after_dd:$after_dd"
It would be nice to rewrite this a bit when we fix it - These are actually checks on free space. This is verifying that free space didn't grow. It would be nice if the test made that clearer.
|
| Comment by Hongchao Zhang [ 05/May/19 ] |
|
this issue is caused by the side effect of previous test, the previous transactions are not committed |
| Comment by Gerrit Updater [ 05/May/19 ] |
|
Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34808 |
| Comment by Gerrit Updater [ 21/May/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34808/ |
| Comment by Peter Jones [ 21/May/19 ] |
|
Landed for 2.13 |
| Comment by Gerrit Updater [ 21/May/19 ] |
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34927 |
| Comment by Gerrit Updater [ 08/Jun/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34927/ |
| Comment by James Nunez (Inactive) [ 15/Nov/19 ] |
|
It looks like we are still experiencing replay-ost-single test 6 failing with the modified error message 'free grew after dd: before:15371264 after_dd:15371264'' . Please see https://testing.whamcloud.com/test_sets/6f49e8d2-07de-11ea-8e77-52540065bddc for one recent failure on b2_13. |
| Comment by Hongchao Zhang [ 16/Nov/19 ] |
|
By searching the fails on Maloo, the new occurrences began at Sept 03, and all are with ZFS backend. |
| Comment by Gerrit Updater [ 18/Nov/19 ] |
|
Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36772 |
| Comment by Gerrit Updater [ 12/Oct/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36772/ |
| Comment by Peter Jones [ 12/Oct/20 ] |
|
Latest fix landed |