[LU-3522] sanity-benchmark test_iozone: "no space left on device" on ZFS - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: Lustre 2.10.0
Affects Version/s: Lustre 2.4.0, Lustre 2.4.1, Lustre 2.5.0, Lustre 2.8.0
Labels:
None
Environment:
server and client : lustre-master build# 1536 zfs

Severity:
3
Rank (Obsolete):
8858

Description

This issue was created by maloo for sarah <sarah@whamcloud.com>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/903643c8-ddca-11e2-85a3-52540035b04c.

The sub-test test_iozone failed with the following error:

iozone (2) failed

test log shows:

Error writing block 7269, fd= 3
write: No space left on device

iozone: interrupted

exiting iozone

 sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (2) failed

Attachments

Issue Links

is blocked by

LU-2049 add support for OBD_CONNECT_GRANT_PARAM

Resolved

is duplicated by

LU-3906 Failure on test suite parallel-scale test_compilebench: IOError, No space left on device

Resolved

LU-4042 Failure on test suite replay-ost-single test_5: no space left

Resolved

mentioned in: Page No Confluence page found with the given URL.; Page No Confluence page found with the given URL.; Page Loading...; Page Loading...; Page Loading...

(3 mentioned in)

Sub-Tasks

Progress

add support for OBD_CONNECT_GRANT_PARAM

Resolved

Nathaniel Clark

Activity

[LU-3522] sanity-benchmark test_iozone: "no space left on device" on ZFS

Sarah Liu added a comment - 08/Jul/15 5:47 AM

I have similar failure in interop testing with master branch ldiskfs

https://testing.hpdd.intel.com/test_sets/b4f55f66-250b-11e5-8009-5254006e85c2

Sarah Liu added a comment - 08/Jul/15 5:47 AM I have similar failure in interop testing with master branch ldiskfs https://testing.hpdd.intel.com/test_sets/b4f55f66-250b-11e5-8009-5254006e85c2

Jian Yu added a comment - 13/Dec/13 8:28 AM

More instances on Lustre b2_4 branch:
https://maloo.whamcloud.com/test_sets/bc739fb2-6353-11e3-8c76-52540035b04c
https://maloo.whamcloud.com/test_sets/75b2103c-6366-11e3-8ae4-52540035b04c
https://maloo.whamcloud.com/test_sets/06fb5b2e-627d-11e3-a8fd-52540035b04c

Jian Yu added a comment - 13/Dec/13 8:28 AM More instances on Lustre b2_4 branch: https://maloo.whamcloud.com/test_sets/bc739fb2-6353-11e3-8c76-52540035b04c https://maloo.whamcloud.com/test_sets/75b2103c-6366-11e3-8ae4-52540035b04c https://maloo.whamcloud.com/test_sets/06fb5b2e-627d-11e3-a8fd-52540035b04c

Jian Yu added a comment - 26/Nov/13 7:31 AM

Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/58/
Distro/Arch: RHEL6.4/x86_64

FSTYPE=zfs
MDSCOUNT=1
MDSSIZE=2097152
OSTCOUNT=2
OSTSIZE=2097152

sanity-benchmark test iozone failed again:
https://maloo.whamcloud.com/test_sets/f6ed7dd6-5604-11e3-8e94-52540035b04c

Jian Yu added a comment - 26/Nov/13 7:31 AM Lustre Build: http://build.whamcloud.com/job/lustre-b2_4/58/ Distro/Arch: RHEL6.4/x86_64 FSTYPE=zfs MDSCOUNT=1 MDSSIZE=2097152 OSTCOUNT=2 OSTSIZE=2097152 sanity-benchmark test iozone failed again: https://maloo.whamcloud.com/test_sets/f6ed7dd6-5604-11e3-8e94-52540035b04c

Jian Yu added a comment - 04/Nov/13 2:31 AM

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/
Distro/Arch: RHEL6.4/x86_64

FSTYPE=zfs
MDSCOUNT=1
MDSSIZE=2097152
OSTCOUNT=2
OSTSIZE=2097152

sanity-benchmark test iozone failed as follows:

                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         3844920     512
Error writing block 7508, fd= 3

iozone: interrupted

exiting iozone

 sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed

Dmesg on client node showed that:

Lustre: DEBUG MARKER: min OST has 2014336kB available, using 3844920kB file size
LustreError: 17272:0:(vvp_io.c:1088:vvp_io_commit_write()) Write page 961062 of inode ffff88007af981f8 failed -28

Maloo report: https://maloo.whamcloud.com/test_sets/51ec0c9c-444f-11e3-8472-52540035b04c

Jian Yu added a comment - 04/Nov/13 2:31 AM Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/ Distro/Arch: RHEL6.4/x86_64 FSTYPE=zfs MDSCOUNT=1 MDSSIZE=2097152 OSTCOUNT=2 OSTSIZE=2097152 sanity-benchmark test iozone failed as follows: random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 3844920 512 Error writing block 7508, fd= 3 iozone: interrupted exiting iozone sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed Dmesg on client node showed that: Lustre: DEBUG MARKER: min OST has 2014336kB available, using 3844920kB file size LustreError: 17272:0:(vvp_io.c:1088:vvp_io_commit_write()) Write page 961062 of inode ffff88007af981f8 failed -28 Maloo report: https://maloo.whamcloud.com/test_sets/51ec0c9c-444f-11e3-8472-52540035b04c

Jian Yu added a comment - 02/Nov/13 2:42 PM

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs

The same failure occurred:
https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c

Jian Yu added a comment - 02/Nov/13 2:42 PM Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/ Distro/Arch: RHEL6.4/x86_64 FSTYPE=zfs The same failure occurred: https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c

Jian Yu added a comment - 02/Nov/13 2:30 PM

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/
Distro/Arch: RHEL6.4/x86_64(server), SLES11SP2/x86_64(client)
FSTYPE=ldiskfs

The same failure occurred:
https://maloo.whamcloud.com/test_sets/e5731cd8-432f-11e3-8676-52540035b04c

Jian Yu added a comment - 02/Nov/13 2:30 PM Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/ Distro/Arch: RHEL6.4/x86_64(server), SLES11SP2/x86_64(client) FSTYPE=ldiskfs The same failure occurred: https://maloo.whamcloud.com/test_sets/e5731cd8-432f-11e3-8676-52540035b04c

Peter Jones added a comment - 19/Sep/13 4:34 PM

Lai

Are you able to assist with this one?

Peter

Peter Jones added a comment - 19/Sep/13 4:34 PM Lai Are you able to assist with this one? Peter

Nathaniel Clark added a comment - 18/Sep/13 12:16 PM - edited

The current state of this bug seems to be: Behaving as expected, wait for OBD_CONNECT_GRANT_PARAM to be supported on the client (~~LU-2049~~).

Nathaniel Clark added a comment - 18/Sep/13 12:16 PM - edited The current state of this bug seems to be: Behaving as expected, wait for OBD_CONNECT_GRANT_PARAM to be supported on the client ( LU-2049 ).

Jian Yu added a comment - 09/Sep/13 3:34 AM

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=ldiskfs

sanity-benchmark test iozone also hit the same failure:
https://maloo.whamcloud.com/test_sets/497a00b4-182b-11e3-b39a-52540035b04c

Jian Yu added a comment - 09/Sep/13 3:34 AM Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2) Distro/Arch: RHEL6.4/x86_64 FSTYPE=ldiskfs sanity-benchmark test iozone also hit the same failure: https://maloo.whamcloud.com/test_sets/497a00b4-182b-11e3-b39a-52540035b04c

Sarah Liu added a comment - 08/Sep/13 8:39 PM

lustre-master build #1652 also hit this issue on ldiskfs:

https://maloo.whamcloud.com/test_sets/c30260ee-15b4-11e3-8938-52540035b04c

Sarah Liu added a comment - 08/Sep/13 8:39 PM lustre-master build #1652 also hit this issue on ldiskfs: https://maloo.whamcloud.com/test_sets/c30260ee-15b4-11e3-8938-52540035b04c

People

Assignee:: Nathaniel Clark

Reporter:: Maloo

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 27/Jun/13 9:28 PM

Updated:: 09/Oct/17 7:55 PM

Resolved:: 09/Oct/17 7:55 PM