https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7106 Lustre <p>Clients are getting error when writing to Lustre server (build 2.7.56):</p> <p>commands like "cp" will return "no space left on device" error.<br/> Here are the corresponding logs:</p> <p>Sep 4 10:06:16 oasis-dm1 kernel: LustreError: 4891:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 9477610 of inode ffff8803d965b138 failed -28<br/> Sep 4 10:06:16 oasis-dm1 kernel: LustreError: 4891:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message<br/> Sep 4 10:19:39 oasis-dm1 kernel: LustreError: 5492:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 804864 of inode ffff88049e96abb8 failed -28<br/> Sep 4 10:19:39 oasis-dm1 kernel: LustreError: 5492:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 3 previous similar messages<br/> Sep 4 10:41:32 oasis-dm1 kernel: LustreError: 7446:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 8473626 of inode ffff88016af646b8 failed -28<br/> Sep 4 10:41:32 oasis-dm1 kernel: LustreError: 7446:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 6 previous similar messages<br/> Sep 4 12:00:54 oasis-dm1 kernel: LustreError: 17162:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 3805354 of inode ffff880940c2cb38 failed -28<br/> Sep 4 12:00:54 oasis-dm1 kernel: LustreError: 17162:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message<br/> Sep 4 12:04:37 oasis-dm1 kernel: LustreError: 17541:0:(vvp_io.c:1081:vvp_io_commit_write()) Write page 6265883 of inode ffff880254611138 failed -28<br/> Sep 4 12:04:37 oasis-dm1 kernel: LustreError: 17541:0:(vvp_io.c:1081:vvp_io_commit_write()) Skipped 1 previous similar message</p> <p>OST/MDT are not lack space/inode ( avail ~16TB / 10+ million on average), checked from client with</p> <p>grep '<span class="error">&#91;0-9&#93;</span>' /proc/fs/lustre/osc/*/kbytes</p> {free,avail,total}<br/> grep '<span class="error">&#91;0-9&#93;</span>' /proc/fs/lustre/osc/*/files{free,total}<br/> grep '<span class="error">&#91;0-9&#93;</span>' /proc/fs/lustre/mdc/*/kbytes{free,avail,total} <p>grep '<span class="error">&#91;0-9&#93;</span>' /proc/fs/lustre/mdc/*/files</p> {free,total} client:<br/> lustre-client-2.5.3-2.6.32_431.29.2.el6.x86_64.x86_64<br/> lustre-client-source-2.4.3-2.6.32_431.20.3.el6.x86_64_gfbfbc94.x86_64<br/> lustre-client-modules-2.5.3-2.6.32_431.29.2.el6.x86_64.x86_64<br/> <br/> server:<br/> <br/> lustre-osd-zfs-mount-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-iokit-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-modules-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-osd-zfs-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-tests-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-source-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> lustre-2.7.56-3.10.73_1.el6.elrepo.x86_64_g1ef0185.x86_64<br/> LU-7106

Lustre client fail with error vvp_io.c:1081:vvp_io_commit_write()) even went there are space in OST and MDT

Bug Major Resolved Incomplete Yang Sheng Haisong Cai Fri, 4 Sep 2015 19:13:19 +0000 Tue, 20 Oct 2015 00:57:45 +0000 Tue, 20 Oct 2015 00:57:45 +0000 Lustre 2.7.0 0 5 <p>A strace taken from an application to an OST:</p> <p>read(3, "\305\251z:8\36]?\216\17\273\203v{\267Yo4\207\347g\227{\7#7\37~#\17\26v"..., 4194304) = 4194304<br/> write(4, "\305\251z:8\36]?\216\17\273\203v{\267Yo4\207\347g\227{\7#7\37~#\17\26v"..., 4194304) = 2985984<br/> write(4, "[\345&amp;\247\201?\377&gt;\371\205/\0\277~\220\10\206\300\252\221\1,OT \350w\26\355\254\213\301"..., 1208320) = -1 ENOSPC (No space left on device)</p> <p>Yet</p> <p>lfs getstripe NA19240.chrom6.SOLID.bfast.YRI.high_coverage.20100311.bam <br/> NA19240.chrom6.SOLID.bfast.YRI.high_coverage.20100311.bam<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_pattern: 1<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: 1<br/> obdidx objid objid group<br/> 1 2862322 0x2bacf2 0</p> <p>panda-OST0001_UUID 28497036288 10651272192 17823039488 37% /oasis/scratch/comet<span class="error">&#91;OST:1&#93;</span></p> <p>Yang Sheng</p> <p>Could you please help with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Haisong,</p> <p>Looks this is a zfs backend. So could you tell the zfs version? I was confused by your server kernel version. It has a 'el6' name but with a '3.10.73' version number. I would very appreciated if you can provided lustre debuglog while the issue hit on. Both server &amp; client is best. </p> <p>Thanks,<br/> YangSheng</p> <p>Hi Haisong,</p> <p>Could you please give us a status update for this ticket? Does it still need further work or should we close it?</p> <p>Thanks,<br/> YangSheng</p> <p>Haisong,</p> <p>I am marking this one as resolved/incomplete. If you would prefer that we do some more work on this issue, just let us know, and provide the information that Yang Sheng has asked for above and we will try to make more progress.</p> <p>Many thanks,<br/> ~ jfc.</p> Development Rank 1|hzxmon: Rank (Obsolete) 9223372036854775807 Severity