[LU-7510] (vvp_io.c:1088:vvp_io_commit_write()) Write page 962977 of inode ffff880fbea44b78 failed -28

• Type: Bug
• Resolution: Done
• Priority: Major
• Affects Version/s: Lustre 2.5.3
• Labels: llnl
• Environment: Servers and clients: 2.5.4-11chaos-11chaos--PRISTINE-2.6.32-573.7.1.1chaos.ch5.4.x86_64; ZFS back end
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Ruth Klundt (Inactive)

Description

We have some production apps and rsync processes failing writes with ENOSPC errors on the ZFS backed FS only. It is currently at ~79%. There are no server side errors, -28 errors as above appeare in the client logs.

I see that LU-3522 and LU-2049 may have a bearing on this issue, is there a 2.5 backport or equivalent fix available?

Issue Links

• is related to: LU-8007 Kernel: LustreError: 191208:0:(vvp_io.c:1086:vvp_io_commit_write()) Write page 82782 of inode ffff8803a8fd06b8 failed -28
• is related to: LU-2049 add support for OBD_CONNECT_GRANT_PARAM

Comments

[John Fuchs-Chesney (Inactive)]

Thanks Ruth.

~ jfc.

[Ruth Klundt (Inactive)]

The file system usage has been reduced to ~70%, and we haven't seen -28 issues or LBUGs since then.

You can close this one, we'll consider the fix for -28 issues to be upgrade to 2.8 lustre on the servers at some point in the future.

If the LBUG re-occurs I'll open a new ticket.

Thanks,
Ruth

[Nathaniel Clark]

The LBUG in question hasn't been changed, though the grant code has been reworked (a la LU-2049) upstream. The negative grant resulting in LBUG should be separate bug, though it's probably 2.5 only.

[Ruth Klundt (Inactive)]

And a specific question, is the LBUG likely addressed by changes upstream or should this be a separate ticket?

[Ruth Klundt (Inactive)]

Nearly all OSS nodes on this file system became inaccessible yesterday, 3 of them showed the LBUG at ofd_grant.c:352:ofd_grant_incoming with negative grant values. I disabled the automated grant release workaround in case it is related to this occurence. The OSTs are 77-79% full at the moment. After that another OSS went down with the same LBUG.

This coincides with the addition of a new cluster, but we haven't done any I/O from it so far, just mounting. Any advice/thoughts?

[Ruth Klundt (Inactive)]

Each of the osts have shown a couple of decreases, in the 3.8-3.9 T range.

[Ruth Klundt (Inactive)]

after deactivating the osts on that node, the rate of increase is slower, but it still is much larger than all the others and not decreasing so far at about ~3.7T.