On April 6, 2016, we noticed one particular file system getting many of the following errors, which corresponded with user job failures:

Kernel: LustreError: 191208:0:(vvp_io.c:1086:vvp_io_commit_write()) Write page 82782 of inode ffff8803a8fd06b8 failed -28

'lfs df' showed the filesystem was only 76% full; however, it also showed that 32 of the 80 OSTs were 89%-90% full. Since deactivating those 32 "near-full" OSTs on 4/6, we haven't seen the problem on that file system.

Consequently, we are now seeing the issue on another file system where OSTs are ~ 90% full.