[LU-3906] Failure on test suite parallel-scale test_compilebench: IOError, No space left on device

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Affects Version/s: Lustre 2.4.1, Lustre 2.5.0, Lustre 2.6.0
• Environment: server and client: lustre-master build #1652
• Severity: 3
• Assignee: Jian Yu
• Reporter: Sarah Liu

Description

https://maloo.whamcloud.com/test_sets/70ec74de-15b9-11e3-8938-52540035b04c

client console shows:

11:00:39:Lustre: DEBUG MARKER: == parallel-scale test compilebench: compilebench == 11:00:31 (1378317631)
11:00:39:Lustre: DEBUG MARKER: /usr/sbin/lctl mark free space=1194928, reducing initial dirs to 1
11:00:40:Lustre: DEBUG MARKER: free space=1194928, reducing initial dirs to 1
11:00:40:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 1         -r 2 --makej
11:00:40:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 1 -r 2 --makej
11:08:26:LustreError: 8551:0:(vvp_io.c:1078:vvp_io_commit_write()) Write page 3250 of inode ffff88001916d1b8 failed -28
11:08:28:LustreError: 8551:0:(vvp_io.c:1078:vvp_io_commit_write()) Write page 3250 of inode ffff88001916d1b8 failed -28

OST console shows:

11:00:42:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 1         -r 2 --makej
11:00:43:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 1 -r 2 --makej
11:05:07:LustreError: 20425:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0004: cli lustre-OST0004_UUID/ffff880021d8d000 left 44863488 < tot_grant 47737472 unstable 0 pending 0
11:05:07:LustreError: 20425:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 6 previous similar messages
11:05:07:LustreError: 21683:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0004: cli 009dd603-5497-a62c-77c6-19fda0311814/ffff880021d8ec00 left 44863488 < tot_grant 47735680 unstable 0 pending 0

Issue Links

• duplicates: LU-3522 sanity-benchmark test_iozone: "no space left on device" on ZFS
• is duplicated by: LU-3912 Failure on test suite sanity-quota test_12a: dd failed
• is duplicated by: LU-3913 Failure on test suite recovery-mds-scale test_failover_mds
• is related to: LU-3909 Interop 2.4.0<->2.5 failure on test suite sanity test_129: return code 28 received instead of expected 27
• is related to: LU-3904 Failure on test suite parallel-scale test_compilebench: IOError, No space left on device
• is related to: LU-3905 Failure on test suite parallel-scale test_compilebench: IOError, No space left on device