[LU-4265] replay-ost-single test_6: space grew after dd (or didn't change)

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.3, Lustre 2.9.0, Lustre 2.12.0, Lustre 2.13.0, Lustre 2.12.1
• Labels: zfs
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:
http://maloo.whamcloud.com/test_sets/627ba06c-48b0-11e3-bdb5-52540035b04c
http://maloo.whamcloud.com/test_sets/600e6472-4f68-11e3-84d3-52540035b04c

The sub-test test_6 failed with the following error:

space grew after dd: before:77312128 after_dd:77312128

Info required for matching: replay-ost-single 6

Issue Links

• is related to: LU-12232 replay-ost-single test 6 fails with ''space grew after dd: before:13442048 after_dd:13442048''

Comments

[James Nunez (Inactive)]

It looks like this test is failing again/still. I’ve gone back to January 2018 and looks at all the times replay-ost-single test 6 failed with the ''space grew after dd:” error and we’ve seen six failures all for ZFS testing:
2018-08-17 - 2.11.53.69 - https://testing.whamcloud.com/test_sets/24be3e5c-a273-11e8-8853-52540065bddc
2019-02-26 – 2.12.0.1 - https://testing.whamcloud.com/test_sets/60e5eab4-3964-11e9-8f69-52540065bddc
2019-04-17 – 2.12.52.85 - https://testing.whamcloud.com/test_sets/4fd6a78e-6170-11e9-9720-52540065bddc
2019-04-19 – 2.12.52.89 - https://testing.whamcloud.com/test_sets/043e5b44-62fd-11e9-aeec-52540065bddc
2019-04-23 – 2.12.52.95 - https://testing.whamcloud.com/test_sets/bf51ebce-65f7-11e9-a6f9-52540065bddc
2019-04-23 – 2.12.1 RC1 - https://testing.whamcloud.com/test_sets/d53ee48a-665d-11e9-8bb1-52540065bddc

Some of the failures look questionable or, possibly, the error message should be modified. For example, when “before” and “after” are the same value:
''space grew after dd: before:13442048 after_dd:13442048''

[James Nunez (Inactive)]

Another failure on master:
2015-12-26 18:38:36 - https://testing.hpdd.intel.com/test_sets/cc75f8cc-ac0a-11e5-8114-5254006e85c2

[Jian Yu]

More failure instance on master branch:
https://testing.hpdd.intel.com/test_sets/94e59696-3640-11e5-84a9-5254006e85c2

[James Nunez (Inactive)]

Another two failures on master in review-zfs-part-2:
2015-07-03 07:45:48 - https://testing.hpdd.intel.com/test_sets/ceba821c-2161-11e5-a979-5254006e85c2
2015-07-14 21:23:00 - https://testing.hpdd.intel.com/test_sets/234cf58c-2a7a-11e5-96c0-5254006e85c2

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/92cbbd1a-16d1-11e5-8436-5254006e85c2

[Johann Lombardi (Inactive)]

Another instance on master:
https://testing.hpdd.intel.com/test_sets/b07629a6-86a6-11e4-b678-5254006e85c2

[Jian Yu]

More failure instance on Lustre b2_5 branch:
https://testing.hpdd.intel.com/test_sets/cdfb95ec-7dd7-11e4-aa98-5254006e85c2

[Isaac Huang (Inactive)]

Another one:
https://testing.hpdd.intel.com/test_sets/197aed5c-592e-11e4-8f95-5254006e85c2

It appeared to be related to LU-3455. The test_6 tried to get before=$(kbytesfree) after file removal and some wait:

    rm -f $f
    sync && sleep 5 && sync  # wait for delete thread

    # wait till space is returned, following
    # (( $before > $after_dd)) test counting on that
    wait_mds_ost_sync || return 4
    wait_destroy_complete || return 5

    local before=$(kbytesfree)

I'd doubt the wait can work reliably with ZFS - sometimes frees can be delayed for a few transaction groups' time. It seemed inherently unreliable to free and wait and get free space for ZFS.

[nasf (Inactive)]

Another failure instance:
https://testing.hpdd.intel.com/test_sets/3e9f4124-45c3-11e4-9397-5254006e85c2

[Jian Yu]

One more instance on Lustre b2_5 branch: https://testing.hpdd.intel.com/test_sets/cd0bb8ee-44dc-11e4-bb5a-5254006e85c2

[Jian Yu]

While verifying patches http://review.whamcloud.com/11541, http://review.whamcloud.com/11411, http://review.whamcloud.com/9318 on Lustre b2_5 branch with FSTYPE=zfs, the same failure occurred:
https://testing.hpdd.intel.com/test_sets/160d2586-31e7-11e4-92e0-5254006e85c2
https://testing.hpdd.intel.com/test_sets/228a2e1e-27b8-11e4-893b-5254006e85c2
https://testing.hpdd.intel.com/test_sets/cb2bd86a-9a50-11e3-965c-52540035b04c