[LU-15353] OST: error destroying object, Failed to write client lcd Created: 09/Dec/21  Updated: 10/Dec/21  Resolved: 10/Dec/21

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Dneg (Inactive) Assignee: Andreas Dilger
Resolution: Not a Bug Votes: 0
Labels: None

Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

Due to a hardware fault, we lost power on a disk array that was part of an OSS. The disk groups in the RAID report as recovered, and an fsck was run, but we are now getting errors, e.g.,:

 Dec  9 12:32:20 coss5 kernel: LustreError: 23370:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a6dc07:0x0]: -30
Dec  9 12:32:20 coss5 kernel: LustreError: 23370:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 15 previous similar messages
Dec  9 12:44:39 coss5 kernel: LustreError: 23479:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x59ebd23:0x0]: -30
Dec  9 12:44:39 coss5 kernel: LustreError: 23479:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 26 previous similar messages
Dec  9 12:54:54 coss5 kernel: LustreError: 21968:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a10caf:0x0]: -30
Dec  9 12:54:54 coss5 kernel: LustreError: 21968:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 113 previous similar messages
Dec  9 13:02:16 coss5 kernel: LustreError: 22016:0:(tgt_lastrcvd.c:1133:tgt_client_del()) charlie-OST0054: failed to update server data, skip client 40af1999-fab1-c3b9-87e0-a070dbf6eb50 zeroing, rc -30
Dec  9 13:05:10 coss5 kernel: LustreError: 22500:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a6f63d:0x0]: -30
Dec  9 13:05:10 coss5 kernel: LustreError: 22500:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 4 previous similar messages
Dec  9 13:08:20 coss5 kernel: LustreError: 12705:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:08:58 coss5 kernel: LustreError: 22537:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:09:23 coss5 kernel: LustreError: 22015:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:09:48 coss5 kernel: LustreError: 22535:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:10:13 coss5 kernel: LustreError: 19790:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:10:38 coss5 kernel: LustreError: 22014:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:11:03 coss5 kernel: LustreError: 12704:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:11:53 coss5 kernel: LustreError: 22004:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:11:53 coss5 kernel: LustreError: 22004:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 1 previous similar message
Dec  9 13:13:08 coss5 kernel: LustreError: 22048:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:13:08 coss5 kernel: LustreError: 22048:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 2 previous similar messages
Dec  9 13:15:39 coss5 kernel: LustreError: 22014:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:15:39 coss5 kernel: LustreError: 22014:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 5 previous similar messages
Dec  9 13:18:45 coss5 kernel: LustreError: 12704:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a0c6ae:0x0]: -30
Dec  9 13:18:45 coss5 kernel: LustreError: 12704:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 14 previous similar messages
Dec  9 13:20:15 coss5 kernel: LustreError: 12704:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:20:15 coss5 kernel: LustreError: 12704:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 10 previous similar messages
Dec  9 13:29:02 coss5 kernel: LustreError: 22001:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:29:02 coss5 kernel: LustreError: 22001:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 20 previous similar messages
Dec  9 13:29:03 coss5 kernel: LustreError: 22490:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a1cc60:0x0]: -30
Dec  9 13:29:03 coss5 kernel: LustreError: 22490:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 660 previous similar messages
Dec  9 13:39:04 coss5 kernel: LustreError: 12699:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:39:04 coss5 kernel: LustreError: 12699:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 23 previous similar messages
Dec  9 13:40:09 coss5 kernel: LustreError: 23370:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x5a0c9dd:0x0]: -30
Dec  9 13:40:09 coss5 kernel: LustreError: 23370:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 39 previous similar messages
Dec  9 13:49:06 coss5 kernel: LustreError: 22492:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:49:06 coss5 kernel: LustreError: 22492:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 23 previous similar messages
Dec  9 13:51:10 coss5 kernel: LustreError: 12702:0:(ofd_dev.c:1804:ofd_destroy_hdl()) charlie-OST0054: error destroying object [0x100540000:0x3a33c87:0x0]: -30
Dec  9 13:51:10 coss5 kernel: LustreError: 12702:0:(ofd_dev.c:1804:ofd_destroy_hdl()) Skipped 2 previous similar messages
Dec  9 13:59:08 coss5 kernel: LustreError: 12693:0:(tgt_lastrcvd.c:1027:tgt_client_new()) charlie-OST0054: Failed to write client lcd at idx 3, rc -30
Dec  9 13:59:08 coss5 kernel: LustreError: 12693:0:(tgt_lastrcvd.c:1027:tgt_client_new()) Skipped 23 previous similar messages

Can you please tell me what the steps are to resolve this issue

Campbell

 Comments   
Comment by Andreas Dilger [ 09/Dec/21 ]

-30 = -EROFS, so it looks like the filesystem was mounted read-only due to the previous error. That would have been posted in the logs before the filesystem turned read-only.

The OST needs to be unmounted, run e2fsck, and remounted.

Comment by Dneg (Inactive) [ 10/Dec/21 ]

Thanks Andreas. I think I made the mistake of running -p rather than -y on my fscks before I opened this ticket. All good now

Generated at Sat Feb 10 03:17:35 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.