[LU-7106] Lustre client fail with error vvp_io.c:1081:vvp_io_commit_write()) even went there are space in OST and MDT

• Type: Bug
• Resolution: Incomplete
• Priority: Major
• Affects Version/s: Lustre 2.7.0

Environment

client:
lustre-client-2.5.3-2.6.32_431.29.2.el6.x86_64.x86_64
lustre-client-source-2.4.3-2.6.32_431.20.3.el6.x86_64_gfbfbc94.x86_64
lustre-client-modules-2.5.3-2.6.32_431.29.2.el6.x86_64.x86_64

server:

lustre-osd-zfs-mount-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-iokit-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-modules-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-osd-zfs-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-tests-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-source-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64
lustre-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64

• Severity: 4
• Assignee: Yang Sheng
• Reporter: Haisong Cai (Inactive)

Description

Clients are getting error when writing to Lustre server (build 2.7.56):

commands like "cp" will return "no space left on device" error.
Here are the corresponding logs:

Sep 4 10:06:16 oasis-dm1 kernel: LustreError: 4891:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 9477610 of inode ffff8803d965b138 failed -28
Sep 4 10:06:16 oasis-dm1 kernel: LustreError: 4891:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message
Sep 4 10:19:39 oasis-dm1 kernel: LustreError: 5492:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 804864 of inode ffff88049e96abb8 failed -28
Sep 4 10:19:39 oasis-dm1 kernel: LustreError: 5492:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 3 previous similar messages
Sep 4 10:41:32 oasis-dm1 kernel: LustreError: 7446:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 8473626 of inode ffff88016af646b8 failed -28
Sep 4 10:41:32 oasis-dm1 kernel: LustreError: 7446:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 6 previous similar messages
Sep 4 12:00:54 oasis-dm1 kernel: LustreError: 17162:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 3805354 of inode ffff880940c2cb38 failed -28
Sep 4 12:00:54 oasis-dm1 kernel: LustreError: 17162:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message
Sep 4 12:04:37 oasis-dm1 kernel: LustreError: 17541:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 6265883 of inode ffff880254611138 failed -28
Sep 4 12:04:37 oasis-dm1 kernel: LustreError: 17541:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message

OST/MDT are not lack space/inode ( avail ~16TB / 10+ million on average), checked from client with

grep '[0-9]' /proc/fs/lustre/osc/*/kbytes

{free,avail,total}
grep '[0-9]' /proc/fs/lustre/osc/*/files{free,total}
grep '[0-9]' /proc/fs/lustre/mdc/*/kbytes{free,avail,total}

grep '[0-9]' /proc/fs/lustre/mdc/*/files

{free,total}