[LU-3522] sanity-benchmark test_iozone: "no space left on device" on ZFS

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1, Lustre 2.5.0, Lustre 2.8.0
• Environment: server and client : lustre-master build# 1536 zfs
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/903643c8-ddca-11e2-85a3-52540035b04c.

The sub-test test_iozone failed with the following error:

iozone (2) failed

test log shows:

Error writing block 7269, fd= 3
write: No space left on device

iozone: interrupted

exiting iozone

 sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (2) failed 

Issue Links

• is blocked by: LU-2049 add support for OBD_CONNECT_GRANT_PARAM
• is duplicated by: LU-3906 Failure on test suite parallel-scale test_compilebench: IOError, No space left on device
• is duplicated by: LU-4042 Failure on test suite replay-ost-single test_5: no space left

Comments

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/
Distro/Arch: RHEL6.4/x86_64(server), SLES11SP2/x86_64(client)
FSTYPE=ldiskfs

The same failure occurred:
https://maloo.whamcloud.com/test_sets/e5731cd8-432f-11e3-8676-52540035b04c

[Peter Jones]

Lai

Are you able to assist with this one?

Peter

[Nathaniel Clark]

The current state of this bug seems to be: Behaving as expected, wait for OBD_CONNECT_GRANT_PARAM to be supported on the client (LU-2049).

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/45/ (2.4.1 RC2)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=ldiskfs

sanity-benchmark test iozone also hit the same failure:
https://maloo.whamcloud.com/test_sets/497a00b4-182b-11e3-b39a-52540035b04c

[Sarah Liu]

lustre-master build #1652 also hit this issue on ldiskfs:

https://maloo.whamcloud.com/test_sets/c30260ee-15b4-11e3-8938-52540035b04c

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/44/ (2.4.1 RC1)
Distro/Arch: RHEL6.4/x86_64
FSTYPE=zfs

sanity-benchmark test iozone also hit the same failure:
https://maloo.whamcloud.com/test_sets/e004601a-1556-11e3-8938-52540035b04c

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/44/ (2.4.1 RC1)
Distro/Arch: RHEL6.4/x86_64 + FC18/x86_64 (Server + Client)
FSTYPE=ldiskfs

sanity-benchmark test iozone hit the same failure:
https://maloo.whamcloud.com/test_sets/becb9218-14ef-11e3-ac48-52540035b04c

[Andreas Dilger]

I put a brief description of what needs to be done into LU-2049. I thought that Jinshan and Johann had worked up a more detailed design for this, but I can't find it.

[Nathaniel Clark]

http://review.whamcloud.com/7402

[Nathaniel Clark]

The difference between ldiskfs and zfs seems to be in the calculated grant request space, namely ofd_grant_from_cli()'s conversion factor for ldiskfs is 1, whereas for zfs it's 32. This is because the ofd_grant_compat() assumes that if the ofd_blockbits (set from statfs which gives the largest available block size in zfs) is greater than 4KB (12 bits) that it should assume the worst case, and shift out by that amount (which in this case is 5 bits).

[Nathaniel Clark]

It appears that when doing O_DIRECT, the transfer runs out of grant space before completing and thus gets an ENOSPC error back (this is zfs only).