Feb 14 22:09:22 sns-oss4 kernel: [132425.563619] LDISKFS-fs error (device dm-1): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 19519corrupted: 3547 blocks free in bitmap, 14336 - in gd
Feb 14 22:09:22 sns-oss4 kernel: [132425.577846]
Feb 14 22:09:22 sns-oss4 kernel: [132425.577848] Aborting journal on device dm-1-8.
Feb 14 22:09:22 sns-oss4 kernel: [132425.605253] LDISKFS-fs (dm-1): Remounting filesystem read-only
Feb 14 22:09:22 sns-oss4 kernel: [132425.649459] LDISKFS-fs error (device dm-1) in ldiskfs_ext_new_extent_cb: Journal has aborted
Feb 14 22:09:22 sns-oss4 kernel: [132425.657967] LustreError: 17682:0:(fsfilt-ldiskfs.c:1327:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (128 bytes)
Feb 14 22:09:22 sns-oss4 kernel: [132425.657972] LustreError: 17682:0:(filter.c:198:filter_finish_transno()) wrote trans 68719477696 for client 0e123953-93bc-fff8-6f8e-cee672729f4a at #4: err = -30
Feb 14 22:09:22 sns-oss4 kernel: [132425.657977] LustreError: 17682:0:(filter_io_26.c:534:filter_direct_io()) can't close transaction: -30
Feb 14 22:09:22 sns-oss4 kernel: [132425.658081] LustreError: 17884:0:(obd.h:1399:obd_transno_commit_cb()) snsfs-OST0003: transno 68719477696 commit error: 2
Feb 14 22:09:23 sns-oss4 kernel: [132426.767066] LustreError: 17684:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:23 sns-oss4 kernel: [132426.767076] LustreError: 17684:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:25 sns-oss4 kernel: [132428.630956] LDISKFS-fs warning (device dm-1): kmmpd: kmmpd being stopped since filesystem has been remounted as readonly.
Feb 14 22:09:25 sns-oss4 kernel: [132428.763531] LustreError: 17685:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:25 sns-oss4 kernel: [132428.763537] LustreError: 17685:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:28 sns-oss4 kernel: [132431.758210] LustreError: 17686:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:28 sns-oss4 kernel: [132431.758220] LustreError: 17686:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:32 sns-oss4 kernel: [132435.751074] LustreError: 17669:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:32 sns-oss4 kernel: [132435.751082] LustreError: 17669:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:37 sns-oss4 kernel: [132440.742516] LustreError: 17676:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:37 sns-oss4 kernel: [132440.742526] LustreError: 17676:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:43 sns-oss4 kernel: [132446.732015] LustreError: 17691:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:43 sns-oss4 kernel: [132446.732022] LustreError: 17691:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:50 sns-oss4 kernel: [132453.719357] LustreError: 17692:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:09:58 sns-oss4 kernel: [132461.705468] LustreError: 17694:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:09:58 sns-oss4 kernel: [132461.705477] LustreError: 17694:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) Skipped 1 previous similar message
Feb 14 22:09:58 sns-oss4 kernel: [132461.705483] LustreError: 17694:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:10:07 sns-oss4 kernel: [132470.689703] LustreError: 17701:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:10:17 sns-oss4 kernel: [132480.671890] LustreError: 17663:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 309 credits: rc = -30
Feb 14 22:10:17 sns-oss4 kernel: [132480.671896] LustreError: 17663:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) Skipped 1 previous similar message
Feb 14 22:10:17 sns-oss4 kernel: [132480.671901] LustreError: 17663:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:10:53 sns-oss4 kernel: [132516.998803] LustreError: 17700:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 45 credits: rc = -30
Feb 14 22:10:53 sns-oss4 kernel: [132516.998813] LustreError: 17700:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:10:55 sns-oss4 kernel: [132518.600234] LustreError: 17683:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:08 sns-oss4 kernel: [132531.204579] LustreError: 17707:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:14 sns-oss4 kernel: [132537.566671] LustreError: 17638:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:20 sns-oss4 kernel: [132543.556099] LustreError: 17708:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:25 sns-oss4 kernel: [132548.205114] LustreError: 17698:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:31 sns-oss4 kernel: [132554.536970] LustreError: 17642:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:38 sns-oss4 kernel: [132561.524631] LustreError: 17706:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:46 sns-oss4 kernel: [132569.510574] LustreError: 17615:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:11:55 sns-oss4 kernel: [132578.494675] LustreError: 17713:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:05 sns-oss4 kernel: [132588.477046] LustreError: 17696:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) can't get handle for 45 credits: rc = -30
Feb 14 22:12:05 sns-oss4 kernel: [132588.477054] LustreError: 17696:0:(fsfilt-ldiskfs.c:501:fsfilt_ldiskfs_brw_start()) Skipped 9 previous similar messages
Feb 14 22:12:05 sns-oss4 kernel: [132588.477061] LustreError: 17696:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:05 sns-oss4 kernel: [132588.478390] LustreError: 17662:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:10 sns-oss4 kernel: [132593.468239] LustreError: 17718:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:26 sns-oss4 kernel: [132609.440115] LustreError: 17705:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:29 sns-oss4 kernel: [132612.434854] LustreError: 17709:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:33 sns-oss4 kernel: [132616.427812] LustreError: 17723:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:38 sns-oss4 kernel: [132621.418810] LustreError: 17715:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:44 sns-oss4 kernel: [132627.408532] LustreError: 17644:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:51 sns-oss4 kernel: [132634.396227] LustreError: 17728:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:12:59 sns-oss4 kernel: [132642.381846] LustreError: 17608:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:13:08 sns-oss4 kernel: [132651.365960] LustreError: 17714:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:13:18 sns-oss4 kernel: [132661.348514] LustreError: 17720:0:(filter_io_26.c:705:filter_commitrw_write()) error starting transaction: rc = -30
Feb 14 22:48:19 sns-oss4 kernel: [134758.760583] Lustre: Failing over snsfs-OST0003
Feb 14 22:48:19 sns-oss4 kernel: [134758.760777] LustreError: 18023:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8107ec056c00 x1426872811394462/t0 o401->@NET_0x2000080dbf921_UUID:17/18 lens 224/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Feb 14 22:48:20 sns-oss4 kernel: [134759.889798] LustreError: 18540:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107)  req@ffff810805dc7c00 x1423518949769300/t0 o13-><?>@<?>:0/0 lens 192/0 e 0 to 0 dl 1360900111 ref 1 fl Interpret:/0/0 rc -107/0
Feb 14 22:48:20 sns-oss4 kernel: [134759.889808] LustreError: 18540:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27 previous similar messages
Feb 14 22:48:20 sns-oss4 kernel: [134759.890419] LustreError: 137-5: UUID 'snsfs-OST0003_UUID' is not available for connect (stopping)
Feb 14 22:48:20 sns-oss4 kernel: [134759.890426] LustreError: Skipped 27 previous similar messages
Feb 14 22:48:21 sns-oss4 kernel: [134760.585264] LustreError: 18400:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107)  req@ffff8103e3d5c000 x1421631118350617/t0 o400-><?>@<?>:0/0 lens 192/0 e 0 to 0 dl 1360900121 ref 1 fl Interpret:H/0/0 rc -107/0
Feb 14 22:48:21 sns-oss4 kernel: [134760.585273] LustreError: 18400:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5 previous similar messages
Feb 14 22:48:21 sns-oss4 kernel: [134760.603668] LustreError: 137-5: UUID 'snsfs-OST0003_UUID' is not available for connect (stopping)
Feb 14 22:48:21 sns-oss4 kernel: [134760.603675] LustreError: Skipped 2 previous similar messages
Feb 14 22:48:21 sns-oss4 kernel: [134760.757027] Lustre: snsfs-OST0003: shutting down for failover; client state will be preserved.
Feb 14 22:48:21 sns-oss4 kernel: [134760.758404] LustreError: 2854:0:(fsfilt-ldiskfs.c:1327:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (512 bytes)
Feb 14 22:48:21 sns-oss4 kernel: [134760.758411] LustreError: 2854:0:(filter.c:757:filter_update_server_data()) error writing lr_server_data: rc = -30
Feb 14 22:48:21 sns-oss4 kernel: [134760.758415] LustreError: 2854:0:(filter.c:1298:filter_post()) error writing server data: rc = -30
Feb 14 22:48:21 sns-oss4 kernel: [134760.758420] LustreError: 2854:0:(fsfilt-ldiskfs.c:1327:fsfilt_ldiskfs_write_record()) can't start transaction for 37 blocks (8 bytes)
Feb 14 22:48:21 sns-oss4 kernel: [134760.758425] LustreError: 2854:0:(filter.c:785:filter_update_last_objid()) error writing group 0 last objid: rc = -30
Feb 14 22:48:21 sns-oss4 kernel: [134760.758429] LustreError: 2854:0:(filter.c:1304:filter_post()) error writing group 0 lastobjid: rc = -30
Feb 14 22:48:21 sns-oss4 kernel: [134760.775297] Lustre: OST snsfs-OST0003 has stopped.
Feb 14 22:48:21 sns-oss4 kernel: [134760.787459] LDISKFS-fs error (device dm-1): ldiskfs_put_super: Couldn't clean up the journal
Feb 14 22:48:21 sns-oss4 kernel: [134760.838416] Lustre: server umount snsfs-OST0003 complete

[ omitting 'processing error' and 'not available for connect' messages ]

Feb 14 22:49:08 sns-oss4 kernel: [134807.634597] LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Feb 14 22:49:08 sns-oss4 kernel: [134807.634603]
Feb 14 22:49:25 sns-oss4 kernel: [134825.085649] LustreError: 137-5: UUID 'snsfs-OST0003_UUID' is not available for connect (no target)
Feb 14 22:49:25 sns-oss4 kernel: [134825.085655] LustreError: Skipped 63 previous similar messages
Feb 14 22:49:25 sns-oss4 kernel: [134825.085662] LustreError: 18336:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8107d0e4b800 x1420341993029920/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1360900265 ref 1 fl Interpret:/0/0 rc -19/0
Feb 14 22:49:25 sns-oss4 kernel: [134825.085667] LustreError: 18336:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 63 previous similar messages
Feb 14 22:49:38 sns-oss3 kernel: [134839.493240] LustreError: 137-5: UUID 'snsfs-OST0003_UUID' is not available for connect (no target)
Feb 14 22:49:38 sns-oss3 kernel: [134839.493248] LustreError: Skipped 64 previous similar messages
Feb 14 22:49:38 sns-oss3 kernel: [134839.493258] LustreError: 17541:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff81041d117000 x1420341993029950/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1360900278 ref 1 fl Interpret:/0/0 rc -19/0
Feb 14 22:49:38 sns-oss3 kernel: [134839.493265] LustreError: 17541:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 64 previous similar messages
Feb 14 22:49:51 sns-oss4 kernel: [134850.622717] LDISKFS-fs warning (device dm-1): ldiskfs_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Feb 14 22:49:51 sns-oss4 kernel: [134850.622725] LDISKFS-fs warning (device dm-1): ldiskfs_clear_journal_err: Marking fs in need of filesystem check.
Feb 14 22:49:51 sns-oss4 kernel: [134850.652530] LDISKFS-fs (dm-1): warning: mounting fs with errors, running e2fsck is recommended
Feb 14 22:49:51 sns-oss4 kernel: [134850.716399] LDISKFS-fs (dm-1): recovery complete
Feb 14 22:49:51 sns-oss4 kernel: [134850.724626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode