[LU-7611] OSTs become "not healthy"

• Type: Bug
• Resolution: Done
• Priority: Major
• Affects Version/s: Lustre 2.5.3
• Environment: RHEL 6.6; 2.6.32-504.30.3.el6.atlas.x86_64
• Severity: 2
• Assignee: Jian Yu
• Reporter: Jesse Hanley

Description

We had a hard power outage early this morning. After hardware fixes, we were about to mount one of our file systems and it appears to be healthy.

On the other file system, we have 5 OSTs that have reported unhealthy. The file system mounted fine, and clients connected, but soon after, I started receiving alerts. I'm in the process of rebooting the nodes so that I can e2fsck them. On each node that has an unhealthy OST, they have messages similar to the following:

[ 2962.001970] LustreError: 24264:0:(tgt_lastrcvd.c:583:tgt_client_new()) atlas2-OST00ca: Failed to write client lcd at idx 19, rc -30
[ 2962.029196] LustreError: 24264:0:(tgt_lastrcvd.c:583:tgt_client_new()) Skipped 140 previous similar messages
[ 3263.951324] LustreError: 24172:0:(ofd_obd.c:1365:ofd_create()) atlas2-OST00ca: unable to precreate: rc = -30
[ 3263.981018] LustreError: 24172:0:(ofd_obd.c:1365:ofd_create()) Skipped 61 previous similar messages
[ 3562.220443] LustreError: 24251:0:(tgt_lastrcvd.c:583:tgt_client_new()) atlas2-OST00ca: Failed to write client lcd at idx 19, rc -30
[ 3562.252970] LustreError: 24251:0:(tgt_lastrcvd.c:583:tgt_client_new()) Skipped 233 previous similar messages
[ 3874.190928] LustreError: 24281:0:(ofd_obd.c:1365:ofd_create()) atlas2-OST00ca: unable to precreate: rc = -30
[ 3874.219280] LustreError: 24281:0:(ofd_obd.c:1365:ofd_create()) Skipped 62 previous similar messages

Is an e2fsck the proper response in this type of situation?

Thanks,
–
Jesse