Mar 5 00:00:12 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:00:12 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 00:00:18 dc2oss12 kernel: : Lustre: dc2-OST007a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:00:18 dc2oss12 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:00:24 dc2oss08 kernel: : Lustre: 14471:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1393995624 last 1393995624 Mar 5 00:00:24 dc2oss08 kernel: : Lustre: 14471:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 00:00:34 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:00:34 dc2oss13 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 00:00:58 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 00:00:58 dc2oss07 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 00:01:11 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:01:11 dc2oss15 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 00:01:14 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:01:14 dc2oss08 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 00:01:14 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:01:14 dc2oss06 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 00:01:43 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 00:01:43 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 00:01:48 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231975937787, new: 231975937786 replay: 0. see LU-617. Mar 5 00:01:48 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231975938215, new: 231975938214 replay: 0. see LU-617. Mar 5 00:02:12 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 00:02:12 dc2oss05 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 00:02:19 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:02:19 dc2oss10 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:02:18 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 00:02:18 dc2oss04 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 00:02:29 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 00:02:29 dc2oss09 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 00:02:46 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:02:46 dc2oss05 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 00:03:40 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 00:03:40 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 00:03:48 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a4_UUID' is not available for connect (no target) Mar 5 00:03:48 dc2oss15 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:04:01 dc2mds01 kernel: : Lustre: 14542:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x2ae1:0x0]] [valid 68719476736] Mar 5 00:04:01 dc2mds01 kernel: : Lustre: 14542:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 00:04:23 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0004_UUID' is not available for connect (no target) Mar 5 00:04:23 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:05:12 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 00:05:12 dc2oss01 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 00:05:26 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 00:05:26 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:05:31 dc2oss09 kernel: : Lustre: 14492:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1393995931 last 1393995931 Mar 5 00:05:31 dc2oss09 kernel: : Lustre: 14492:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 00:05:22 dc2mds01 kernel: : Lustre: 1374:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x2aec:0x0]] [valid 68719476736] Mar 5 00:05:22 dc2mds01 kernel: : Lustre: 1374:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 00:05:35 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231976590183, new: 231976590182 replay: 0. see LU-617. Mar 5 00:05:54 dc2oss13 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1393995954 last 1393995954 Mar 5 00:06:19 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:05:54 dc2oss13 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 00:06:19 dc2oss03 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 00:06:19 dc2oss03 kernel: : Lustre: 17495:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1393995979 last 1393995979 Mar 5 00:06:19 dc2oss03 kernel: : Lustre: 17495:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 00:05:58 dc2oss15 kernel: : Lustre: 15946:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1393995958 last 1393995958 Mar 5 00:05:58 dc2oss15 kernel: : Lustre: 15946:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 00:06:04 dc2oss16 kernel: : Lustre: 17405:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1393995964 last 1393995964 Mar 5 00:06:04 dc2oss16 kernel: : Lustre: 17405:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 00:06:12 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 00:06:12 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:06:34 dc2oss11 kernel: : Lustre: 15522:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1393995994 last 1393995994 Mar 5 00:06:34 dc2oss11 kernel: : Lustre: 15522:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 00:06:26 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1393995986 last 1393995986 Mar 5 00:06:26 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:06:32 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1393995992 last 1393995992 Mar 5 00:06:32 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 00:06:56 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:06:56 dc2oss02 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 00:06:56 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1393996016 last 1393996016 Mar 5 00:06:56 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 00:06:45 dc2oss07 kernel: : Lustre: 14316:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1393996005 last 1393996005 Mar 5 00:06:45 dc2oss07 kernel: : Lustre: 14316:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:06:55 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 00:06:55 dc2oss06 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 00:06:57 dc2oss06 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1393996017 last 1393996017 Mar 5 00:06:57 dc2oss06 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:07:04 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:07:04 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:07:04 dc2oss04 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 00:07:04 dc2oss04 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 00:06:53 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 00:06:53 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 18 previous similar messages Mar 5 00:07:13 dc2oss10 kernel: : Lustre: 16506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1393996033 last 1393996033 Mar 5 00:07:13 dc2oss10 kernel: : Lustre: 16506:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:07:09 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231976856788, new: 231976856787 replay: 0. see LU-617. Mar 5 00:07:09 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 2 previous similar messages Mar 5 00:07:30 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:07:30 dc2oss16 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:07:30 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1393996050 last 1393996050 Mar 5 00:07:30 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 00:07:36 dc2oss14 kernel: : Lustre: 14586:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1393996056 last 1393996056 Mar 5 00:07:36 dc2oss14 kernel: : Lustre: 14586:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 00:07:44 dc2oss11 kernel: : Lustre: dc2-OST0072: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:07:44 dc2oss11 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:07:42 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:07:42 dc2oss01 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 00:07:42 dc2oss01 kernel: : Lustre: 15308:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1393996062 last 1393996062 Mar 5 00:07:42 dc2oss01 kernel: : Lustre: 15308:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 00:08:07 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:08:07 dc2oss07 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 00:08:31 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:08:31 dc2oss14 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:08:31 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 00:08:31 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 00:08:28 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 00:08:28 dc2mds01 kernel: : Lustre: Skipped 22 previous similar messages Mar 5 00:08:28 dc2mds01 kernel: : Lustre: 27966:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1393996108 last 1393996108 Mar 5 00:08:28 dc2mds01 kernel: : Lustre: 27966:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 22 previous similar messages Mar 5 00:08:47 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 00:08:47 dc2oss08 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:10:00 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 00:10:00 dc2oss14 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 00:10:19 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:10:19 dc2oss12 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 00:10:26 dc2oss08 kernel: : Lustre: 15898:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1393996226 last 1393996226 Mar 5 00:10:26 dc2oss08 kernel: : Lustre: 15898:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 00:10:32 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009a_UUID' is not available for connect (no target) Mar 5 00:10:32 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 00:10:31 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:10:31 dc2oss09 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 00:10:41 dc2oss13 kernel: : Lustre: dc2-OST007e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:10:41 dc2oss13 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 00:11:14 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 00:11:14 dc2oss07 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 00:11:15 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:11:15 dc2oss15 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 00:11:17 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:11:17 dc2oss06 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 00:11:23 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:11:23 dc2oss08 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 00:11:51 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 00:11:51 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 00:12:19 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:12:19 dc2oss10 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:12:18 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 00:12:18 dc2oss05 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:12:38 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0063_UUID' is not available for connect (no target) Mar 5 00:12:38 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:12:50 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:12:50 dc2oss05 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 00:13:07 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 00:13:07 dc2oss04 kernel: : LustreError: Skipped 12 previous similar messages Mar 5 00:13:46 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 00:13:46 dc2oss03 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:14:03 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a0_UUID' is not available for connect (no target) Mar 5 00:14:03 dc2oss15 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 00:14:42 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 00:14:42 dc2oss02 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 00:14:37 dc2mds01 kernel: : Lustre: 11138:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f962:0x12f:0x0]] [valid 68719476736] Mar 5 00:14:37 dc2mds01 kernel: : Lustre: 11138:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 2 previous similar messages Mar 5 00:15:03 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231978202390, new: 231978202389 replay: 0. see LU-617. Mar 5 00:15:14 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 00:15:14 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:15:32 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1393996532 last 1393996532 Mar 5 00:15:32 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 00:15:35 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 00:15:35 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:15:55 dc2oss13 kernel: : Lustre: 16055:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1393996555 last 1393996555 Mar 5 00:15:55 dc2oss13 kernel: : Lustre: 16055:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:15:58 dc2oss15 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1393996558 last 1393996558 Mar 5 00:15:58 dc2oss15 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 00:16:20 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:16:20 dc2oss03 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 00:16:20 dc2oss03 kernel: : Lustre: 17089:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1393996580 last 1393996580 Mar 5 00:16:20 dc2oss03 kernel: : Lustre: 17089:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 00:16:14 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1393996574 last 1393996574 Mar 5 00:16:14 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 00:16:27 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1393996587 last 1393996587 Mar 5 00:16:27 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 00:16:40 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1393996600 last 1393996600 Mar 5 00:16:40 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 00:16:35 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1393996595 last 1393996595 Mar 5 00:16:35 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 00:16:50 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0048: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0d048000 cur 1393996610 last 1393996610 Mar 5 00:16:50 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:17:09 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:17:09 dc2oss02 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 00:17:09 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1393996629 last 1393996629 Mar 5 00:17:09 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 00:17:05 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1393996625 last 1393996625 Mar 5 00:17:05 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 00:17:16 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1393996636 last 1393996636 Mar 5 00:17:16 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 00:17:10 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:17:10 dc2oss04 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 00:17:24 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 00:17:24 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 00:17:30 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:17:30 dc2oss16 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 00:17:34 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1393996654 last 1393996654 Mar 5 00:17:34 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 00:17:36 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1393996656 last 1393996656 Mar 5 00:17:36 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 00:17:44 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:17:44 dc2oss11 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 00:17:46 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:17:46 dc2oss01 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 00:17:46 dc2oss01 kernel: : Lustre: 15346:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1393996666 last 1393996666 Mar 5 00:17:46 dc2oss01 kernel: : Lustre: 15346:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 00:17:44 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 00:17:44 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 25 previous similar messages Mar 5 00:18:10 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:18:10 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:18:18 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 00:18:18 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 00:18:07 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231978709125, new: 231978709124 replay: 0. see LU-617. Mar 5 00:18:31 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231978773406, new: 231978773405 replay: 0. see LU-617. Mar 5 00:18:54 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 00:18:54 dc2oss08 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:18:56 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 00:18:56 dc2oss12 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:18:59 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:18:59 dc2oss14 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 00:19:06 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 00:19:06 dc2mds01 kernel: : Lustre: Skipped 27 previous similar messages Mar 5 00:19:06 dc2mds01 kernel: : Lustre: 27966:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1393996746 last 1393996746 Mar 5 00:19:06 dc2mds01 kernel: : Lustre: 27966:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 27 previous similar messages Mar 5 00:20:06 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0081_UUID' is not available for connect (no target) Mar 5 00:20:06 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 00:20:26 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:20:26 dc2oss12 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 00:20:33 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1393996833 last 1393996833 Mar 5 00:20:33 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 00:20:34 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 00:20:34 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:20:35 dc2oss09 kernel: : Lustre: dc2-OST005c: Client 458d3df7-f3d6-ee35-8f08-f667a18d1919 (at 149.165.229.232@tcp) reconnecting Mar 5 00:20:35 dc2oss09 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 00:20:47 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:20:47 dc2oss13 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 00:21:16 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:21:16 dc2oss15 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 00:21:20 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:21:20 dc2oss06 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 00:21:29 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:21:29 dc2oss08 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 00:21:42 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 00:21:42 dc2oss07 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 00:22:03 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0056_UUID' is not available for connect (no target) Mar 5 00:22:03 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 00:22:18 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 00:22:18 dc2oss05 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 00:22:19 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:22:19 dc2oss10 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:22:49 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 00:22:49 dc2oss09 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:22:52 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:22:52 dc2oss05 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 00:23:34 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 00:23:34 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:24:03 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 00:24:03 dc2oss03 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 00:24:07 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 00:24:07 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 00:25:29 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 00:25:29 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 00:25:35 dc2oss09 kernel: : Lustre: 16391:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1393997135 last 1393997135 Mar 5 00:25:35 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1393997135 last 1393997135 Mar 5 00:25:35 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:25:29 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 00:25:29 dc2oss01 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:25:59 dc2oss13 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1393997159 last 1393997158 Mar 5 00:25:59 dc2oss13 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 00:26:22 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:26:22 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:26:22 dc2oss03 kernel: : Lustre: 16650:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1393997182 last 1393997182 Mar 5 00:26:22 dc2oss03 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:26:22 dc2oss03 kernel: : Lustre: 16650:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:26:04 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1393997164 last 1393997164 Mar 5 00:26:04 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 00:26:15 dc2oss16 kernel: : Lustre: 17320:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1393997175 last 1393997175 Mar 5 00:26:15 dc2oss16 kernel: : Lustre: 17320:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 00:26:32 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1393997192 last 1393997192 Mar 5 00:26:32 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 00:26:42 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1393997202 last 1393997202 Mar 5 00:26:42 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 00:26:37 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1393997197 last 1393997197 Mar 5 00:26:37 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 00:26:56 dc2oss07 kernel: : Lustre: 15970:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1393997216 last 1393997216 Mar 5 00:26:56 dc2oss07 kernel: : Lustre: 15970:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 00:27:16 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:27:16 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:27:16 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1393997236 last 1393997236 Mar 5 00:27:16 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 00:27:15 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1393997235 last 1393997235 Mar 5 00:27:15 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:27:18 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1393997238 last 1393997238 Mar 5 00:27:18 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 00:27:34 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:27:34 dc2oss16 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 00:27:25 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:27:25 dc2oss04 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:27:43 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 00:27:43 dc2oss06 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 00:27:45 dc2oss14 kernel: : Lustre: 15506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1393997265 last 1393997265 Mar 5 00:27:45 dc2oss14 kernel: : Lustre: 15506:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:27:45 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1393997265 last 1393997265 Mar 5 00:27:45 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 00:28:00 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 00:28:00 dc2oss13 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 00:27:56 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:27:56 dc2oss11 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 00:27:58 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:27:58 dc2oss01 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 00:27:58 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1393997278 last 1393997278 Mar 5 00:27:58 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 00:28:15 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:28:15 dc2oss07 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:28:29 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 00:28:29 dc2oss11 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 00:29:00 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 00:29:00 dc2oss12 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:29:06 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:29:06 dc2oss14 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 00:29:13 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 00:29:13 dc2oss08 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 00:29:16 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:29:16 dc2mds01 kernel: : Lustre: Skipped 16 previous similar messages Mar 5 00:29:16 dc2mds01 kernel: : Lustre: 10017:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1393997356 last 1393997356 Mar 5 00:29:16 dc2mds01 kernel: : Lustre: 10017:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages Mar 5 00:30:13 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 00:30:13 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 9 previous similar messages Mar 5 00:30:28 dc2oss12 kernel: : Lustre: dc2-OST007a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:30:28 dc2oss12 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 00:30:36 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:30:36 dc2oss09 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 00:30:43 dc2oss09 kernel: : Lustre: dc2-OST005e: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 00:30:44 dc2oss09 kernel: : LustreError: 22637:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8805dd6f4c00 x1461680444420547/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1393997474 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 00:30:44 dc2oss09 kernel: : LustreError: 22637:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 00:30:44 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 00:30:44 dc2oss09 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 00:30:36 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1393997436 last 1393997436 Mar 5 00:30:36 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 00:30:38 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 00:30:38 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 00:30:49 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:30:49 dc2oss13 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 00:30:51 dc2oss11 kernel: : Lustre: dc2-OST006c: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 00:30:51 dc2oss11 kernel: : LustreError: 8818:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807f2779400 x1461680444420548/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1393997486 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 00:30:51 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 00:30:52 dc2oss11 kernel: : LustreError: 14685:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8801a1a54400 x1461680444420559/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1393997487 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 00:30:52 dc2oss11 kernel: : LustreError: 14685:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 00:30:52 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 00:30:52 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 00:30:54 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 00:30:54 dc2oss10 kernel: : LustreError: 8806:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd0dc2c00 x1461680444420552/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1393997485 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 00:30:54 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 00:30:54 dc2oss10 kernel: : LustreError: 8743:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fea4f850 x1461680444420581/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1393997488 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 00:30:54 dc2oss10 kernel: : LustreError: 8743:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 00:30:54 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 00:30:54 dc2oss10 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 00:31:17 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:31:17 dc2oss15 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 00:31:23 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:31:23 dc2oss06 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:31:34 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:31:34 dc2oss08 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:31:42 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 00:31:42 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:31:54 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 00:31:54 dc2oss07 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 00:32:15 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 00:32:15 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 00:32:19 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:32:19 dc2oss10 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 00:32:26 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 00:32:26 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128082] Lustre: Skipped 3 previous similar messages Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128052] Lustre: 44805:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669103618117 sent from dc2-OST0041-osc-ffff880956851000 to NID 10.10.0.7@o2ib 7s ago has timed out (7s prior to deadline). Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128058] Lustre: 44805:0:(client.c:1492:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128076] Lustre: dc2-OST0041-osc-ffff880956851000: Connection to service dc2-OST0041 via nid 10.10.0.7@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128424] Lustre: dc2-OST0041-osc-ffff880956851000: Connection restored to service dc2-OST0041 using nid 10.10.0.7@o2ib. Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128426] Lustre: Skipped 3 previous similar messages Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128440] Lustre: Server dc2-OST0041_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 00:32:36 149.165.238.1 kernel: kernel: [4271928.128442] Lustre: Skipped 3 previous similar messages Mar 5 00:32:57 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:32:57 dc2oss05 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 00:33:29 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 00:33:29 dc2oss09 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 00:34:13 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 00:34:13 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:33:46 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 00:33:46 dc2oss04 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.132064] LustreError: 11-0: an error occurred while communicating with 10.10.0.7@o2ib. The obd_ping operation failed with -107 Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.132064] Lustre: dc2-OST0041-osc-ffff880956851000: Connection to service dc2-OST0041 via nid 10.10.0.7@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.132647] LustreError: 167-0: This client was evicted by dc2-OST0041; in progress operations using this service will fail. Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.132690] Lustre: Server dc2-OST0041_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.133549] LustreError: 46315:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace dc2-OST0041-osc-ffff880956851000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.133555] LustreError: 46315:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff8802c6312b00 (22659055/0/0/0) (rc: 1) Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.133571] Lustre: dc2-OST0041-osc-ffff880956851000: Connection restored to service dc2-OST0041 using nid 10.10.0.7@o2ib. Mar 5 00:34:19 149.165.238.1 kernel: kernel: [4272031.133674] LustreError: 44805:0:(lov_request.c:211:lov_update_enqueue_set()) enqueue objid 0x2ed5 subobj 0x159bfef on OST idx 65: rc -5 Mar 5 00:34:13 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 00:34:13 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:34:16 dc2oss07 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 107s: evicting client at 149.165.238.1@tcp ns: filter-dc2-OST0041_UUID lock: ffff8817ded90480/0xe3df804f8159eb79 lrc: 3/0,0 mode: PR/PR res: 22659055/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xe213f1feb640ddea expref: 183 pid: 16104 timeout 9012891539 Mar 5 00:35:38 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1393997738 last 1393997738 Mar 5 00:35:38 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 00:35:47 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 00:35:47 dc2oss01 kernel: : LustreError: Skipped 11 previous similar messages Mar 5 00:36:05 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1393997765 last 1393997765 Mar 5 00:36:05 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 00:36:28 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:36:28 dc2oss03 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:36:28 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1393997788 last 1393997788 Mar 5 00:36:28 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 00:36:16 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1393997776 last 1393997776 Mar 5 00:36:16 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 00:36:16 dc2oss13 kernel: : Lustre: 16037:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1393997776 last 1393997776 Mar 5 00:36:16 dc2oss13 kernel: : Lustre: 16037:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 00:36:39 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1393997799 last 1393997799 Mar 5 00:36:39 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 00:36:44 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1393997804 last 1393997804 Mar 5 00:36:44 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 00:36:41 dc2oss04 kernel: : Lustre: 16678:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1393997801 last 1393997801 Mar 5 00:36:41 dc2oss04 kernel: : Lustre: 16678:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 00:37:06 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12508400 cur 1393997826 last 1393997826 Mar 5 00:37:06 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 00:37:19 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:37:19 dc2oss02 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 00:37:19 dc2oss02 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1393997839 last 1393997839 Mar 5 00:37:19 dc2oss02 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 00:37:19 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 00:37:19 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 00:37:19 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1393997839 last 1393997839 Mar 5 00:37:19 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:37:26 dc2oss06 kernel: : Lustre: 16103:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1393997846 last 1393997846 Mar 5 00:37:26 dc2oss06 kernel: : Lustre: 16103:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 00:37:29 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:37:29 dc2oss04 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:37:45 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1393997865 last 1393997865 Mar 5 00:37:45 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 00:37:51 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:37:51 dc2oss16 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 00:37:54 dc2oss14 kernel: : Lustre: 15670:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1393997874 last 1393997874 Mar 5 00:37:54 dc2oss14 kernel: : Lustre: 15670:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 00:37:59 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:37:59 dc2oss11 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:38:01 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:38:01 dc2oss01 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:38:01 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1393997881 last 1393997881 Mar 5 00:38:01 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 00:38:19 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:38:19 dc2oss07 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 00:38:26 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 00:38:26 dc2oss06 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 00:38:35 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 00:38:35 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 00:38:30 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 00:38:30 dc2oss13 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 00:39:08 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:39:08 dc2oss14 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 00:39:08 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 00:39:08 dc2oss12 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 00:39:34 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 00:39:34 dc2oss08 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 00:39:52 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:39:52 dc2mds01 kernel: : Lustre: Skipped 23 previous similar messages Mar 5 00:39:52 dc2mds01 kernel: : Lustre: 10017:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1393997992 last 1393997992 Mar 5 00:39:52 dc2mds01 kernel: : Lustre: 10017:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 23 previous similar messages Mar 5 00:40:33 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:40:33 dc2oss12 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 00:40:36 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:40:36 dc2oss09 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:40:37 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1393998037 last 1393998037 Mar 5 00:40:37 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 73 previous similar messages Mar 5 00:40:49 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:40:49 dc2oss13 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 00:41:20 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:41:20 dc2oss15 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:41:14 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 00:41:14 dc2oss14 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 00:41:29 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:41:29 dc2oss06 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 00:41:34 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:41:34 dc2oss08 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 00:42:16 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 00:42:16 dc2oss10 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 00:42:23 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:42:23 dc2oss10 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 00:42:28 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 00:42:28 dc2oss16 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 00:42:26 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 00:42:26 dc2oss07 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:42:31 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 00:42:31 dc2oss05 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:42:58 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:42:58 dc2oss05 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 00:44:14 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 00:44:14 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:43:57 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST005f_UUID' is not available for connect (no target) Mar 5 00:43:57 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:44:12 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 00:44:12 dc2oss04 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 00:44:23 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 00:44:23 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 14 previous similar messages Mar 5 00:45:41 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 00:45:41 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:45:48 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1393998348 last 1393998348 Mar 5 00:45:48 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 00:46:09 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1393998369 last 1393998369 Mar 5 00:46:09 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:46:29 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:46:29 dc2oss03 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 00:46:29 dc2oss03 kernel: : Lustre: 16915:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1393998389 last 1393998389 Mar 5 00:46:29 dc2oss03 kernel: : Lustre: 16915:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 00:46:19 dc2oss13 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1393998379 last 1393998379 Mar 5 00:46:19 dc2oss13 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 00:46:23 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1393998383 last 1393998383 Mar 5 00:46:23 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 00:46:15 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 00:46:15 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:46:41 dc2oss04 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1393998401 last 1393998401 Mar 5 00:46:41 dc2oss04 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 00:46:46 dc2oss11 kernel: : Lustre: 15541:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1393998406 last 1393998406 Mar 5 00:46:46 dc2oss11 kernel: : Lustre: 15541:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 00:46:49 dc2oss12 kernel: : Lustre: 16159:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1393998409 last 1393998409 Mar 5 00:46:49 dc2oss12 kernel: : Lustre: 16159:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 00:47:19 dc2oss10 kernel: : Lustre: 16341:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1393998439 last 1393998439 Mar 5 00:47:19 dc2oss10 kernel: : Lustre: 16341:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 00:47:16 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1393998436 last 1393998436 Mar 5 00:47:16 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 00:47:29 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:47:29 dc2oss02 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 00:47:29 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1393998449 last 1393998449 Mar 5 00:47:29 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 00:47:36 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0004_UUID' is not available for connect (no target) Mar 5 00:47:36 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:47:26 dc2oss06 kernel: : Lustre: 15870:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1393998446 last 1393998446 Mar 5 00:47:26 dc2oss06 kernel: : Lustre: 15870:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 00:47:29 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:47:29 dc2oss04 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 00:47:48 dc2oss05 kernel: : Lustre: 15954:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1393998468 last 1393998468 Mar 5 00:47:48 dc2oss05 kernel: : Lustre: 15954:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 00:47:54 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:47:54 dc2oss16 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 00:47:54 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1393998474 last 1393998474 Mar 5 00:47:54 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 00:48:01 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:48:01 dc2oss11 kernel: : Lustre: Skipped 132 previous similar messages Mar 5 00:48:06 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:48:06 dc2oss01 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 00:48:06 dc2oss01 kernel: : Lustre: 15033:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1393998486 last 1393998486 Mar 5 00:48:06 dc2oss01 kernel: : Lustre: 15033:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 00:48:20 dc2oss07 kernel: : Lustre: dc2-OST0044: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:48:20 dc2oss07 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 00:48:27 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 00:48:27 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:48:45 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 00:48:45 dc2oss13 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 00:48:56 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 00:48:56 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 00:49:11 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:49:11 dc2oss14 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 00:49:36 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 00:49:36 dc2oss12 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 00:50:05 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 00:50:05 dc2oss08 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 00:50:07 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 00:50:07 dc2mds01 kernel: : Lustre: Skipped 26 previous similar messages Mar 5 00:50:07 dc2mds01 kernel: : Lustre: 9751:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1393998607 last 1393998607 Mar 5 00:50:07 dc2mds01 kernel: : Lustre: 9751:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 26 previous similar messages Mar 5 00:50:34 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:50:34 dc2oss12 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 00:50:49 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1393998649 last 1393998649 Mar 5 00:50:49 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 00:50:48 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:50:48 dc2oss09 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 00:50:54 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:50:54 dc2oss13 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 00:51:20 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:51:20 dc2oss15 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 00:51:31 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:51:31 dc2oss06 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 00:51:30 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 00:51:30 dc2oss14 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 00:51:39 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:51:39 dc2oss08 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 00:52:23 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 00:52:23 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 00:52:29 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:52:29 dc2oss10 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 00:52:33 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 00:52:33 dc2oss07 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 00:52:37 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 00:52:37 dc2oss05 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 00:52:37 dc2mds01 kernel: : Lustre: 11148:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x5618:0x0]] [valid 68719476736] Mar 5 00:52:37 dc2mds01 kernel: : Lustre: 11148:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 00:52:49 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009a_UUID' is not available for connect (no target) Mar 5 00:52:49 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:52:59 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:52:59 dc2oss05 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 00:53:18 dc2mds01 kernel: : Lustre: 14554:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x5bbf:0x0]] [valid 68719476736] Mar 5 00:53:18 dc2mds01 kernel: : Lustre: 14554:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 00:54:16 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 00:54:16 dc2oss03 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 00:54:06 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST005f_UUID' is not available for connect (no target) Mar 5 00:54:06 dc2oss09 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 00:54:15 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 00:54:15 dc2oss04 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 00:54:48 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 00:54:48 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 18 previous similar messages Mar 5 00:55:57 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 00:55:57 dc2oss15 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 00:55:58 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1393998958 last 1393998958 Mar 5 00:55:58 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 00:56:19 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1393998979 last 1393998979 Mar 5 00:56:19 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 00:56:36 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:56:36 dc2oss03 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 00:56:36 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1393998996 last 1393998996 Mar 5 00:56:36 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 00:56:23 dc2oss13 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1393998983 last 1393998983 Mar 5 00:56:23 dc2oss13 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 00:56:26 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1393998986 last 1393998986 Mar 5 00:56:26 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 00:56:22 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 00:56:22 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 00:56:41 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1393999001 last 1393999001 Mar 5 00:56:41 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 00:57:01 dc2 CROND[20126]: (root) CMD (/export/utilities/lustre_ost_capacity.sh) Mar 5 00:56:53 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1393999013 last 1393999013 Mar 5 00:56:53 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 00:56:56 dc2oss11 kernel: : Lustre: 15422:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1393999016 last 1393999016 Mar 5 00:56:56 dc2oss11 kernel: : Lustre: 15422:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 00:57:30 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1393999050 last 1393999050 Mar 5 00:57:30 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 00:57:37 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 00:57:37 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 00:57:40 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:57:40 dc2oss02 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 00:57:40 dc2oss02 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1393999060 last 1393999060 Mar 5 00:57:40 dc2oss02 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 00:57:29 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1393999049 last 1393999049 Mar 5 00:57:29 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 00:57:29 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1393999049 last 1393999049 Mar 5 00:57:29 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 00:57:32 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:57:32 dc2oss04 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 00:57:49 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1393999069 last 1393999069 Mar 5 00:57:49 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 00:57:55 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:57:55 dc2oss16 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 00:58:04 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1393999084 last 1393999083 Mar 5 00:58:04 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 00:58:01 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:58:01 dc2oss11 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 00:58:12 dc2oss01 kernel: : Lustre: dc2-OST0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:58:12 dc2oss01 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 00:58:12 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c003ae000 cur 1393999092 last 1393999092 Mar 5 00:58:12 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 00:58:22 dc2oss07 kernel: : Lustre: dc2-OST0048: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:58:22 dc2oss07 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 00:58:58 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 00:58:58 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:58:57 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 00:58:57 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 00:59:07 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 00:59:07 dc2oss11 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 00:59:14 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 00:59:14 dc2oss14 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 00:59:40 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 00:59:40 dc2oss12 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 00:59:53 dc2mds01 kernel: : Lustre: 10012:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x6eff:0x0]] [valid 68719476736] Mar 5 00:59:53 dc2mds01 kernel: : Lustre: 10012:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 01:00:18 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0047_UUID' is not available for connect (no target) Mar 5 01:00:18 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 01:00:38 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:00:38 dc2oss12 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 01:00:27 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:00:27 dc2mds01 kernel: : Lustre: Skipped 19 previous similar messages Mar 5 01:00:27 dc2mds01 kernel: : Lustre: 11191:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1393999227 last 1393999227 Mar 5 01:00:27 dc2mds01 kernel: : Lustre: 11191:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 19 previous similar messages Mar 5 01:00:49 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:00:49 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 01:00:58 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:00:58 dc2oss13 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 01:01:21 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:01:21 dc2oss15 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 01:01:33 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:01:33 dc2oss06 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 01:01:40 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:01:40 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:01:40 dc2oss08 kernel: : Lustre: Skipped 67 previous similar messages Mar 5 01:01:40 dc2oss08 kernel: : Lustre: Skipped 67 previous similar messages Mar 5 01:01:40 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1393999300 last 1393999300 Mar 5 01:01:40 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 01:02:08 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 01:02:08 dc2oss14 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 01:02:32 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 01:02:32 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 01:02:34 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:02:34 dc2oss10 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 01:02:41 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 01:02:41 dc2oss07 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 01:02:29 dc2mds01 kernel: : Lustre: 5941:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f962:0x2d1:0x0]] [valid 68719476736] Mar 5 01:02:29 dc2mds01 kernel: : Lustre: 5941:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 14 previous similar messages Mar 5 01:02:59 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:02:59 dc2oss05 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:03:09 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 01:03:09 dc2oss16 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 01:03:24 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 01:03:24 dc2oss05 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 01:04:20 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 01:04:20 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 01:04:27 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 01:04:27 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 01:04:54 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 01:04:54 dc2oss09 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 01:05:59 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1393999559 last 1393999558 Mar 5 01:05:59 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 01:06:14 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 01:06:14 dc2oss15 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:06:23 dc2oss15 kernel: : Lustre: 15885:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1393999583 last 1393999583 Mar 5 01:06:23 dc2oss15 kernel: : Lustre: 15885:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 01:06:44 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:06:44 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:06:44 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 01:06:44 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 01:06:44 dc2oss03 kernel: : Lustre: 16917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1393999604 last 1393999604 Mar 5 01:06:44 dc2oss03 kernel: : Lustre: 16917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 01:06:26 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1393999586 last 1393999586 Mar 5 01:06:26 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 01:06:35 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1393999595 last 1393999595 Mar 5 01:06:35 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 01:06:27 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 01:06:27 dc2oss01 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:06:28 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:06:28 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) Skipped 11 previous similar messages Mar 5 01:06:42 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1393999602 last 1393999602 Mar 5 01:06:42 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 01:06:53 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1393999613 last 1393999613 Mar 5 01:06:53 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 01:07:05 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1393999625 last 1393999625 Mar 5 01:07:05 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 01:07:32 dc2oss06 kernel: : Lustre: 14487:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1393999652 last 1393999652 Mar 5 01:07:32 dc2oss06 kernel: : Lustre: 14487:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 01:07:30 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1393999650 last 1393999650 Mar 5 01:07:30 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 01:07:30 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1393999650 last 1393999650 Mar 5 01:07:30 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:07:44 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:07:44 dc2oss02 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 01:07:44 dc2oss02 kernel: : Lustre: 16224:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1393999664 last 1393999664 Mar 5 01:07:44 dc2oss02 kernel: : Lustre: 16224:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 01:07:53 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1393999673 last 1393999673 Mar 5 01:07:53 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 01:08:08 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1393999688 last 1393999688 Mar 5 01:08:08 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 01:07:59 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:07:59 dc2oss16 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 01:07:58 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:07:58 dc2oss04 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 01:08:19 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 01:08:19 dc2oss02 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:08:13 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:08:13 dc2oss11 kernel: : Lustre: Skipped 131 previous similar messages Mar 5 01:08:23 dc2oss07 kernel: : Lustre: dc2-OST0048: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:08:23 dc2oss07 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:08:20 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:08:20 dc2oss01 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 01:08:20 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1393999700 last 1393999700 Mar 5 01:08:20 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 01:08:44 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231987381801, new: 231987381800 replay: 0. see LU-617. Mar 5 01:09:08 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 01:09:08 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 01:09:10 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 01:09:10 dc2oss11 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 01:09:14 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:09:14 dc2oss14 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 01:09:25 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 01:09:25 dc2oss06 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 01:09:41 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 01:09:41 dc2oss12 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 01:10:19 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 01:10:19 dc2oss08 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 01:10:39 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:10:39 dc2oss12 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 01:10:50 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:10:50 dc2oss09 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 01:10:42 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:10:42 dc2mds01 kernel: : Lustre: Skipped 21 previous similar messages Mar 5 01:10:42 dc2mds01 kernel: : Lustre: 5941:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1393999842 last 1393999842 Mar 5 01:10:42 dc2mds01 kernel: : Lustre: 5941:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 21 previous similar messages Mar 5 01:11:04 dc2oss13 kernel: : Lustre: dc2-OST0086: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:04 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:04 dc2oss13 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 01:11:04 dc2oss13 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 01:11:27 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:27 dc2oss15 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 01:11:40 dc2oss08 kernel: : Lustre: dc2-OST004d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:40 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:40 dc2oss08 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 01:11:40 dc2oss08 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 01:11:40 dc2oss08 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1393999900 last 1393999900 Mar 5 01:11:40 dc2oss08 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 01:11:40 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:11:40 dc2oss06 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:12:28 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 01:12:28 dc2oss14 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:12:36 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:12:36 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:12:43 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 01:12:43 dc2oss07 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 01:12:48 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0056_UUID' is not available for connect (no target) Mar 5 01:12:48 dc2oss10 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 01:12:50 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231988053414, new: 231988053413 replay: 0. see LU-617. Mar 5 01:13:02 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:13:02 dc2oss05 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 01:13:10 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231988112125, new: 231988112124 replay: 0. see LU-617. Mar 5 01:13:29 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 01:13:29 dc2oss16 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 01:13:32 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 01:13:32 dc2oss05 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 01:14:30 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 01:14:30 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:14:58 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 01:14:58 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:15:03 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0016_UUID' is not available for connect (no target) Mar 5 01:15:03 dc2oss04 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 01:15:27 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231988521938, new: 231988521937 replay: 0. see LU-617. Mar 5 01:16:00 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c00a7e400 cur 1394000160 last 1394000160 Mar 5 01:16:00 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 01:16:21 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 01:16:21 dc2oss15 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 01:16:30 dc2oss16 kernel: : Lustre: 17368:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394000190 last 1394000190 Mar 5 01:16:30 dc2oss16 kernel: : Lustre: 17368:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 01:16:46 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:16:46 dc2oss03 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:16:46 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0df95000 cur 1394000206 last 1394000206 Mar 5 01:16:46 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 01:16:28 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0096: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c267d7c00 cur 1394000188 last 1394000188 Mar 5 01:16:28 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 01:16:38 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394000198 last 1394000198 Mar 5 01:16:38 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 01:16:39 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 01:16:39 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:16:44 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394000204 last 1394000204 Mar 5 01:16:44 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 01:16:38 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:16:39 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 01:16:55 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394000215 last 1394000215 Mar 5 01:16:55 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:17:08 dc2oss11 kernel: : Lustre: 15541:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394000228 last 1394000228 Mar 5 01:17:08 dc2oss11 kernel: : Lustre: 15541:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:17:34 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394000254 last 1394000254 Mar 5 01:17:34 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 01:17:31 dc2oss10 kernel: : Lustre: 16347:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1394000251 last 1394000251 Mar 5 01:17:31 dc2oss10 kernel: : Lustre: 16347:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 01:17:45 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:17:45 dc2oss02 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 01:17:45 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394000265 last 1394000265 Mar 5 01:17:45 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 01:17:51 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394000271 last 1394000271 Mar 5 01:17:51 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 01:17:54 dc2oss05 kernel: : Lustre: 16114:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394000274 last 1394000274 Mar 5 01:17:54 dc2oss05 kernel: : Lustre: 16114:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 01:18:09 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394000289 last 1394000289 Mar 5 01:18:09 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 01:18:00 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:18:00 dc2oss04 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 01:18:03 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:18:03 dc2oss16 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 01:18:14 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:18:14 dc2oss11 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 01:18:24 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:18:24 dc2oss07 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 01:18:25 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:18:25 dc2oss01 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 01:18:25 dc2oss01 kernel: : Lustre: 15376:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394000305 last 1394000305 Mar 5 01:18:25 dc2oss01 kernel: : Lustre: 15376:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 01:18:46 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 01:18:46 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:19:15 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:19:15 dc2oss14 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 01:19:16 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 01:19:16 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 01:19:33 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 01:19:33 dc2oss11 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 01:19:33 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002f_UUID' is not available for connect (no target) Mar 5 01:19:33 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 01:20:10 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 01:20:10 dc2oss12 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 01:20:36 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0045_UUID' is not available for connect (no target) Mar 5 01:20:36 dc2oss08 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:20:41 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:20:41 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 01:20:51 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:20:51 dc2oss09 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 01:20:49 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 01:20:49 dc2mds01 kernel: : Lustre: Skipped 28 previous similar messages Mar 5 01:20:49 dc2mds01 kernel: : Lustre: 18281:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394000449 last 1394000449 Mar 5 01:20:49 dc2mds01 kernel: : Lustre: 18281:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 28 previous similar messages Mar 5 01:21:09 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:21:09 dc2oss13 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:21:32 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:21:32 dc2oss15 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 01:21:42 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:21:42 dc2oss06 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 01:22:32 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 01:22:32 dc2oss14 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 01:22:40 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:22:40 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:22:55 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:22:55 dc2oss08 kernel: : Lustre: dc2-OST004d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:22:55 dc2oss08 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 01:22:55 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1394000575 last 1394000575 Mar 5 01:22:55 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 86 previous similar messages Mar 5 01:23:00 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 01:23:00 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:23:05 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:23:05 dc2oss05 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 01:23:08 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 01:23:08 dc2oss07 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 01:23:44 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 01:23:44 dc2oss16 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 01:23:58 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 01:23:58 dc2oss05 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:25:01 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 01:25:01 dc2oss03 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 01:25:11 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 01:25:11 dc2oss04 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:25:17 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 01:25:17 dc2oss09 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:26:01 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1394000761 last 1394000761 Mar 5 01:26:01 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:26:53 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:26:53 dc2oss03 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:26:53 dc2oss03 kernel: : Lustre: 17089:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394000813 last 1394000813 Mar 5 01:26:53 dc2oss03 kernel: : Lustre: 17089:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:26:39 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394000799 last 1394000799 Mar 5 01:26:39 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 01:26:40 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 01:26:40 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:26:41 dc2oss15 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from 025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp t0 exp ffff88175d02b800 cur 1394000801 last 1394000801 Mar 5 01:26:41 dc2oss15 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 01:26:41 dc2oss16 kernel: : Lustre: 15749:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from 025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp t0 exp ffff8816b5ba4800 cur 1394000801 last 1394000801 Mar 5 01:26:41 dc2oss16 kernel: : Lustre: 15749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 01:26:40 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 01:26:40 dc2oss01 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 01:26:50 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394000810 last 1394000810 Mar 5 01:26:50 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 01:26:57 dc2oss12 kernel: : Lustre: 16043:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394000817 last 1394000817 Mar 5 01:26:57 dc2oss12 kernel: : Lustre: 16043:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 01:26:55 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:26:55 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) Skipped 21 previous similar messages Mar 5 01:27:14 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394000834 last 1394000834 Mar 5 01:27:14 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 01:27:35 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394000855 last 1394000855 Mar 5 01:27:35 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 01:27:32 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394000852 last 1394000852 Mar 5 01:27:32 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 01:27:50 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:27:50 dc2oss02 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 01:27:50 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394000870 last 1394000870 Mar 5 01:27:50 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:27:54 dc2oss05 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394000874 last 1394000874 Mar 5 01:27:54 dc2oss05 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 01:27:55 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394000875 last 1394000875 Mar 5 01:27:55 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 01:28:07 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:28:07 dc2oss16 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 01:28:12 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:28:12 dc2oss04 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 01:28:17 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394000897 last 1394000897 Mar 5 01:28:17 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 01:28:19 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:28:19 dc2oss11 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 01:28:24 dc2oss07 kernel: : Lustre: dc2-OST0044: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:28:24 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 01:28:53 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 01:28:53 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 01:28:41 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:28:41 dc2oss01 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 01:28:41 dc2oss01 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394000921 last 1394000921 Mar 5 01:28:41 dc2oss01 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:29:25 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:29:25 dc2oss14 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 01:29:34 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 01:29:34 dc2oss06 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:29:47 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 01:29:47 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 01:29:50 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 01:29:50 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:30:12 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 01:30:12 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 01:30:44 dc2oss12 kernel: : Lustre: dc2-OST007a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:30:44 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 01:30:55 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:30:55 dc2oss09 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:31:12 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:31:12 dc2oss13 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 01:31:38 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:31:38 dc2oss15 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:31:44 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:31:44 dc2oss06 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 01:31:55 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:31:55 dc2mds01 kernel: : Lustre: Skipped 39 previous similar messages Mar 5 01:31:55 dc2mds01 kernel: : Lustre: 5943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394001115 last 1394001115 Mar 5 01:31:55 dc2mds01 kernel: : Lustre: 5943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 39 previous similar messages Mar 5 01:32:06 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 01:32:06 dc2oss08 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 01:32:44 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:32:44 dc2oss10 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 01:32:48 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 01:32:48 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:33:05 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0056_UUID' is not available for connect (no target) Mar 5 01:33:05 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 01:33:23 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:33:23 dc2oss05 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 01:33:19 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:33:19 dc2oss08 kernel: : Lustre: Skipped 152 previous similar messages Mar 5 01:33:19 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394001199 last 1394001199 Mar 5 01:33:19 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 152 previous similar messages Mar 5 01:33:25 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004e_UUID' is not available for connect (no target) Mar 5 01:33:25 dc2oss07 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 01:33:54 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 01:33:54 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:34:20 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 01:34:20 dc2oss05 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:35:41 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 01:35:41 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:35:26 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 01:35:26 dc2oss09 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 01:35:30 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 01:35:30 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 01:36:02 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394001362 last 1394001362 Mar 5 01:36:02 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:36:56 dc2oss03 kernel: : Lustre: dc2-OST001a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:36:56 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:36:56 dc2oss03 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:36:56 dc2oss03 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:36:56 dc2oss03 kernel: : Lustre: 17032:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c03192800 cur 1394001416 last 1394001416 Mar 5 01:36:56 dc2oss03 kernel: : Lustre: 17032:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:36:43 dc2oss13 kernel: : Lustre: 16037:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394001403 last 1394001403 Mar 5 01:36:43 dc2oss13 kernel: : Lustre: 16037:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 01:36:43 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394001403 last 1394001403 Mar 5 01:36:43 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 01:36:45 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394001405 last 1394001405 Mar 5 01:36:45 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 01:36:51 dc2oss04 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394001411 last 1394001411 Mar 5 01:36:51 dc2oss04 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:36:55 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 01:36:55 dc2oss15 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:36:47 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 01:36:47 dc2oss01 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 01:36:57 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394001417 last 1394001417 Mar 5 01:36:57 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 01:37:01 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:37:01 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) Skipped 14 previous similar messages Mar 5 01:37:15 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394001435 last 1394001435 Mar 5 01:37:15 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 01:37:57 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:37:57 dc2oss02 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 01:37:57 dc2oss02 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394001477 last 1394001477 Mar 5 01:37:57 dc2oss02 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 01:37:40 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394001460 last 1394001460 Mar 5 01:37:40 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 01:37:35 dc2mds01 kernel: : Lustre: 10012:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f796:0x5f4e:0x0]] [valid 68719476736] Mar 5 01:37:35 dc2mds01 kernel: : Lustre: 10012:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 01:37:47 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394001467 last 1394001467 Mar 5 01:37:47 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 01:37:55 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394001475 last 1394001475 Mar 5 01:37:55 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 01:38:00 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12508400 cur 1394001480 last 1394001480 Mar 5 01:38:00 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 01:38:14 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:38:14 dc2oss16 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 01:38:22 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:38:22 dc2oss11 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 01:38:24 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:38:24 dc2oss07 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:38:23 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:38:23 dc2oss04 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 01:38:29 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394001509 last 1394001509 Mar 5 01:38:29 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 01:39:02 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0005_UUID' is not available for connect (no target) Mar 5 01:39:02 dc2oss02 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:38:44 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:38:44 dc2oss01 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:38:44 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394001524 last 1394001523 Mar 5 01:38:44 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 01:39:39 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:39:39 dc2oss14 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 01:39:43 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002c_UUID' is not available for connect (no target) Mar 5 01:39:43 dc2oss06 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 01:39:51 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 01:39:51 dc2oss13 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 01:40:17 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 01:40:17 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:40:20 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 01:40:20 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 01:40:58 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:40:58 dc2oss09 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:41:09 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:41:09 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 01:41:19 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:41:19 dc2oss13 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 01:41:44 dc2oss15 kernel: : Lustre: dc2-OST0097: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:41:44 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:41:44 dc2oss15 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 01:41:48 dc2oss06 kernel: : Lustre: dc2-OST003c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) reconnecting Mar 5 01:41:48 dc2oss06 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:41:55 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 01:41:55 dc2mds01 kernel: : Lustre: Skipped 22 previous similar messages Mar 5 01:41:55 dc2mds01 kernel: : Lustre: 9751:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394001715 last 1394001715 Mar 5 01:41:55 dc2mds01 kernel: : Lustre: 9751:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 22 previous similar messages Mar 5 01:42:11 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 01:42:11 dc2oss08 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:42:46 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:42:46 dc2oss10 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 01:42:49 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 01:42:49 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:43:11 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 01:43:11 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:43:24 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:43:24 dc2oss05 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 01:43:33 dc2oss08 kernel: : Lustre: dc2-OST0052: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:43:33 dc2oss08 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 01:43:33 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1394001813 last 1394001813 Mar 5 01:43:33 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:43:54 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 01:43:54 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 01:43:54 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 01:43:54 dc2oss07 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 01:44:30 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 01:44:30 dc2oss05 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 01:45:38 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 01:45:38 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 01:45:49 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 01:45:49 dc2oss09 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 01:46:07 dc2oss09 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394001967 last 1394001967 Mar 5 01:46:07 dc2oss09 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 01:47:01 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:47:01 dc2oss03 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 01:47:01 dc2oss03 kernel: : Lustre: 16661:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394002021 last 1394002021 Mar 5 01:47:01 dc2oss03 kernel: : Lustre: 16661:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 01:46:45 dc2oss16 kernel: : Lustre: 17238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394002005 last 1394002005 Mar 5 01:46:45 dc2oss16 kernel: : Lustre: 17238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 01:46:46 dc2oss13 kernel: : Lustre: 16055:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394002006 last 1394002006 Mar 5 01:46:46 dc2oss13 kernel: : Lustre: 16055:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 01:46:50 dc2oss15 kernel: : Lustre: 14553:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394002010 last 1394002010 Mar 5 01:46:50 dc2oss15 kernel: : Lustre: 14553:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 01:46:51 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bffed2000 cur 1394002011 last 1394002011 Mar 5 01:46:51 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 01:47:02 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394002022 last 1394002022 Mar 5 01:47:02 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 01:47:05 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 01:47:05 dc2oss15 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 01:47:21 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394002041 last 1394002041 Mar 5 01:47:21 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 01:47:15 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 01:47:15 dc2oss01 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 01:47:40 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394002060 last 1394002060 Mar 5 01:47:40 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 01:47:47 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394002067 last 1394002067 Mar 5 01:47:47 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 01:48:05 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:05 dc2oss02 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 01:48:05 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394002085 last 1394002085 Mar 5 01:48:05 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 01:47:43 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:47:43 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 27 previous similar messages Mar 5 01:47:57 dc2oss05 kernel: : Lustre: 15803:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394002077 last 1394002077 Mar 5 01:47:57 dc2oss05 kernel: : Lustre: 15803:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 01:48:02 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394002082 last 1394002082 Mar 5 01:48:02 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 01:48:21 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:21 dc2oss16 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 01:48:24 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:24 dc2oss07 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 01:48:31 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:31 dc2oss11 kernel: : Lustre: Skipped 76 previous similar messages Mar 5 01:48:23 dc2oss04 kernel: : Lustre: dc2-OST0029: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:23 dc2oss04 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 01:48:29 dc2oss14 kernel: : Lustre: 15475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394002109 last 1394002109 Mar 5 01:48:29 dc2oss14 kernel: : Lustre: 15475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 01:49:03 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 01:49:03 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 01:49:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 01:49:08 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 01:48:54 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:48:54 dc2oss01 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 01:48:54 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394002133 last 1394002133 Mar 5 01:48:54 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 01:49:44 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:49:44 dc2oss14 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 01:49:50 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 01:49:50 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:50:23 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 01:50:23 dc2oss12 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 01:50:58 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007d_UUID' is not available for connect (no target) Mar 5 01:50:58 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:50:59 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:50:59 dc2oss09 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 01:51:10 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:51:10 dc2oss12 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 01:51:14 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 01:51:14 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:51:34 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:51:34 dc2oss13 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 01:51:45 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:51:45 dc2oss15 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 01:51:49 dc2oss06 kernel: : Lustre: dc2-OST003e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:51:49 dc2oss06 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 01:52:18 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 01:52:18 dc2mds01 kernel: : Lustre: Skipped 42 previous similar messages Mar 5 01:52:18 dc2mds01 kernel: : Lustre: 18281:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394002338 last 1394002338 Mar 5 01:52:18 dc2mds01 kernel: : Lustre: 18281:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 42 previous similar messages Mar 5 01:52:51 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:52:51 dc2oss10 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 01:53:09 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 01:53:09 dc2oss14 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 01:53:25 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:53:25 dc2oss05 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 01:53:34 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:53:34 dc2oss08 kernel: : Lustre: Skipped 56 previous similar messages Mar 5 01:53:34 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394002414 last 1394002414 Mar 5 01:53:34 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 56 previous similar messages Mar 5 01:53:55 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 01:53:55 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 01:53:52 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 01:53:52 dc2oss10 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 01:54:03 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 01:54:03 dc2oss07 kernel: : LustreError: Skipped 9 previous similar messages Mar 5 01:54:07 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 01:54:07 dc2oss16 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 01:54:30 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 01:54:30 dc2oss05 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 01:55:40 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001d_UUID' is not available for connect (no target) Mar 5 01:55:40 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 01:55:57 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 01:55:57 dc2oss09 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 01:56:09 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394002569 last 1394002569 Mar 5 01:56:09 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 01:56:46 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394002606 last 1394002606 Mar 5 01:56:46 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 01:56:50 dc2oss15 kernel: : Lustre: 15835:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394002610 last 1394002610 Mar 5 01:56:50 dc2oss15 kernel: : Lustre: 15835:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 01:57:13 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:57:13 dc2oss03 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 01:57:13 dc2oss03 kernel: : Lustre: 17080:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1394002633 last 1394002633 Mar 5 01:57:13 dc2oss03 kernel: : Lustre: 17080:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 01:56:52 dc2oss13 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394002612 last 1394002612 Mar 5 01:56:52 dc2oss13 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 01:57:06 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1394002626 last 1394002626 Mar 5 01:57:06 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 01:57:08 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394002628 last 1394002628 Mar 5 01:57:08 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 01:57:14 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 01:57:14 dc2oss15 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 01:57:25 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394002645 last 1394002645 Mar 5 01:57:25 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 01:57:29 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 01:57:29 dc2oss01 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 01:57:40 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394002660 last 1394002660 Mar 5 01:57:40 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 01:57:49 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394002669 last 1394002669 Mar 5 01:57:49 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 01:58:06 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:58:06 dc2oss02 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 01:58:06 dc2oss02 kernel: : Lustre: 16190:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394002686 last 1394002686 Mar 5 01:58:06 dc2oss02 kernel: : Lustre: 16190:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 01:57:59 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394002679 last 1394002679 Mar 5 01:57:59 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 01:58:10 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0048: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0d048000 cur 1394002690 last 1394002690 Mar 5 01:58:10 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 01:58:27 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:58:27 dc2oss16 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 01:58:31 dc2oss11 kernel: : Lustre: dc2-OST0070: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 01:58:31 dc2oss11 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 01:58:29 dc2oss07 kernel: : Lustre: dc2-OST0045: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 01:58:29 dc2oss07 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 01:58:24 dc2oss04 kernel: : Lustre: dc2-OST0023: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 01:58:24 dc2oss04 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 01:58:34 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394002714 last 1394002714 Mar 5 01:58:34 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 01:58:38 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 01:58:38 dc2oss10 kernel: : LustreError: 1173:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880659868800 x1461679898185212/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394002752 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 01:58:38 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 01:58:51 dc2oss02 kernel: : Lustre: dc2-OST0011: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 01:58:51 dc2oss02 kernel: : LustreError: 21036:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803bf15c000 x1461679898185204/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394002761 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 01:58:51 dc2oss02 kernel: : LustreError: 21036:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 01:58:51 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 01:58:51 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 01:58:51 dc2oss02 kernel: : LustreError: 16058:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88078f9c0400 x1461679898185259/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394002766 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 01:58:51 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 01:59:01 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:59:01 dc2oss01 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 01:59:01 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394002741 last 1394002741 Mar 5 01:59:01 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 01:59:48 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 01:59:48 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 01:59:23 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 01:59:23 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 35 previous similar messages Mar 5 01:59:45 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 01:59:45 dc2oss14 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 02:00:07 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002a_UUID' is not available for connect (no target) Mar 5 02:00:07 dc2oss06 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:00:28 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 02:00:28 dc2oss12 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 02:00:56 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 02:00:56 dc2oss02 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 02:00:28 dc2mds01 kernel: : Lustre: 1369:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f962:0x12f:0x0]] [valid 68719476736] Mar 5 02:00:28 dc2mds01 kernel: : Lustre: 1369:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 02:00:59 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:00:59 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 02:01:16 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 02:01:16 dc2oss13 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:01:27 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:01:27 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:01:27 dc2oss12 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 02:01:27 dc2oss12 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 02:01:37 dc2oss13 kernel: : Lustre: dc2-OST0086: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:01:37 dc2oss13 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 02:01:48 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:01:48 dc2oss15 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 02:01:56 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:01:56 dc2oss06 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 02:02:21 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) Mar 5 02:02:21 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:02:32 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:02:32 dc2mds01 kernel: : Lustre: Skipped 36 previous similar messages Mar 5 02:02:32 dc2mds01 kernel: : Lustre: 1369:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394002952 last 1394002952 Mar 5 02:02:32 dc2mds01 kernel: : Lustre: 1369:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 36 previous similar messages Mar 5 02:02:52 dc2oss10 kernel: : Lustre: dc2-OST0061: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:02:52 dc2oss10 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 02:03:09 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 231996726278, new: 231996726277 replay: 0. see LU-617. Mar 5 02:03:25 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 02:03:25 dc2oss14 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 02:03:30 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:03:30 dc2oss05 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 02:03:45 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:03:45 dc2oss08 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 02:03:45 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394003025 last 1394003025 Mar 5 02:03:45 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 02:03:52 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 02:03:52 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:03:58 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 02:03:58 dc2oss08 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 02:04:25 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 02:04:25 dc2oss16 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:04:41 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 02:04:41 dc2oss07 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 02:05:16 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 02:05:16 dc2oss05 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 02:05:57 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 02:05:57 dc2oss04 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 02:06:19 dc2oss09 kernel: : Lustre: 16169:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25e45c00 cur 1394003179 last 1394003179 Mar 5 02:06:19 dc2oss09 kernel: : Lustre: 16169:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 02:06:22 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 02:06:22 dc2oss09 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:06:46 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394003206 last 1394003206 Mar 5 02:06:46 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 02:07:14 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:07:14 dc2oss03 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 02:07:14 dc2oss03 kernel: : Lustre: 16910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1394003234 last 1394003234 Mar 5 02:07:14 dc2oss03 kernel: : Lustre: 16910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 02:06:58 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394003218 last 1394003218 Mar 5 02:06:58 dc2oss13 kernel: : Lustre: 15856:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394003218 last 1394003218 Mar 5 02:06:58 dc2oss13 kernel: : Lustre: 15856:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 02:06:50 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394003210 last 1394003210 Mar 5 02:06:50 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 02:07:06 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394003226 last 1394003226 Mar 5 02:07:06 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 02:07:09 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394003229 last 1394003229 Mar 5 02:07:09 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:07:27 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394003247 last 1394003247 Mar 5 02:07:27 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 02:07:44 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394003264 last 1394003264 Mar 5 02:07:44 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 02:07:54 dc2oss06 kernel: : Lustre: 14424:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394003274 last 1394003274 Mar 5 02:07:54 dc2oss06 kernel: : Lustre: 14424:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 02:08:08 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:08 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:08 dc2oss02 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 02:08:08 dc2oss02 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 02:08:08 dc2oss02 kernel: : Lustre: 16155:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394003288 last 1394003288 Mar 5 02:08:08 dc2oss02 kernel: : Lustre: 16155:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 02:07:59 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394003279 last 1394003279 Mar 5 02:07:59 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 02:07:52 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 02:07:52 dc2oss01 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 02:08:19 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394003299 last 1394003299 Mar 5 02:08:19 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 02:08:32 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:32 dc2oss11 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 02:08:31 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:31 dc2oss07 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 02:08:34 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394003314 last 1394003314 Mar 5 02:08:34 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 02:08:33 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:33 dc2oss04 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 02:08:40 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:08:40 dc2oss16 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 02:09:01 dc2oss01 kernel: : Lustre: dc2-OST0005: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:09:01 dc2oss01 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 02:09:01 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0005: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37715128 exp ffff880be6c96400 cur 1394003341 last 1394003341 Mar 5 02:09:01 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 02:09:23 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 02:09:23 dc2oss15 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 02:09:51 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 02:09:51 dc2oss03 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 02:09:57 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:09:57 dc2oss14 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 02:09:48 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 02:09:48 dc2mds01 kernel: : LustreError: 9751:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 02:10:14 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 02:10:14 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:10:32 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 02:10:32 dc2oss12 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:11:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 02:11:08 dc2oss02 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 02:10:59 dc2oss09 kernel: : Lustre: dc2-OST0057: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:10:59 dc2oss09 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 02:11:29 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:11:29 dc2oss12 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 02:11:40 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:11:40 dc2oss13 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 02:11:56 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:11:56 dc2oss15 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 02:12:05 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:12:05 dc2oss06 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 02:12:37 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 02:12:37 dc2oss13 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:12:40 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 02:12:40 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:12:59 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:12:59 dc2oss10 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 02:13:30 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:13:30 dc2oss05 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 02:13:47 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:13:47 dc2oss08 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 02:13:47 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394003627 last 1394003627 Mar 5 02:13:47 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 02:13:48 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 02:13:48 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 02:13:52 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 02:13:52 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:13:59 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0045_UUID' is not available for connect (no target) Mar 5 02:13:59 dc2oss08 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 02:13:59 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 02:13:59 dc2mds01 kernel: : Lustre: Skipped 26 previous similar messages Mar 5 02:13:59 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394003639 last 1394003639 Mar 5 02:13:59 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 26 previous similar messages Mar 5 02:14:42 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 02:14:42 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 02:14:51 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009c_UUID' is not available for connect (no target) Mar 5 02:14:51 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:16:22 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394003782 last 1394003782 Mar 5 02:16:22 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 02:16:27 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 02:16:27 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 02:16:46 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 104192, left: 104192, last alive is 24 secs ago Mar 5 02:16:46 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 104192, left: 104192, last alive is 24 secs ago Mar 5 02:16:48 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394003808 last 1394003808 Mar 5 02:16:48 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 02:16:59 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394003819 last 1394003819 Mar 5 02:16:59 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 02:16:51 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394003811 last 1394003811 Mar 5 02:16:51 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 02:17:29 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:17:29 dc2oss03 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 02:17:29 dc2oss03 kernel: : Lustre: 17536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394003849 last 1394003849 Mar 5 02:17:29 dc2oss03 kernel: : Lustre: 17536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 02:17:11 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394003831 last 1394003831 Mar 5 02:17:11 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 02:17:12 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 02:17:12 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:17:10 dc2oss04 kernel: : Lustre: 16714:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394003830 last 1394003830 Mar 5 02:17:10 dc2oss04 kernel: : Lustre: 16714:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 02:17:17 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001e_UUID' is not available for connect (no target) Mar 5 02:17:17 dc2oss04 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 02:17:30 dc2oss11 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394003850 last 1394003850 Mar 5 02:17:30 dc2oss11 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 02:17:47 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc10e800 cur 1394003867 last 1394003867 Mar 5 02:17:47 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 02:18:00 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1394003880 last 1394003880 Mar 5 02:18:00 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 02:18:10 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:18:10 dc2oss02 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 02:18:10 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394003890 last 1394003890 Mar 5 02:18:10 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 02:18:02 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 02:18:02 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:18:18 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394003898 last 1394003898 Mar 5 02:18:18 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 02:18:34 dc2oss14 kernel: : Lustre: 15564:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394003914 last 1394003914 Mar 5 02:18:34 dc2oss14 kernel: : Lustre: 15564:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 02:18:34 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:18:34 dc2oss07 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:18:34 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394003914 last 1394003914 Mar 5 02:18:34 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 02:18:36 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:18:36 dc2oss11 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 02:18:33 dc2oss04 kernel: : Lustre: dc2-OST0022: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:18:33 dc2oss04 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 02:18:49 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:18:49 dc2oss16 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 02:19:05 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 744032, left: 744032, last alive is 8 secs ago Mar 5 02:19:05 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 744032, left: 744032, last alive is 8 secs ago Mar 5 02:19:03 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:19:03 dc2oss01 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:19:03 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394003943 last 1394003943 Mar 5 02:19:03 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 02:19:15 dc2oss12 kernel: : Lustre: Service thread pid 32759 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:19:15 dc2oss12 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:19:15 dc2oss12 kernel: : LustreError: dumping log to /tmp/lustre-log.1394003955.32759 Mar 5 02:19:16 dc2oss12 kernel: : Lustre: Service thread pid 14612 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:19:16 dc2oss12 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:19:16 dc2oss12 kernel: : LustreError: dumping log to /tmp/lustre-log.1394003956.14612 Mar 5 02:19:23 dc2oss12 kernel: : Lustre: Service thread pid 14569 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:19:23 dc2oss12 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:19:23 dc2oss12 kernel: : LustreError: dumping log to /tmp/lustre-log.1394003963.14569 Mar 5 02:19:26 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 02:19:26 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:19:50 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 02:19:50 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 14 previous similar messages Mar 5 02:20:07 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:20:07 dc2oss14 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 02:20:25 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 02:20:25 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:20:52 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 02:20:52 dc2oss03 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 02:20:46 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 02:20:46 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 02:20:59 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:20:59 dc2oss09 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 02:21:38 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 02:21:38 dc2oss02 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:21:27 dc2oss08 kernel: : Lustre: Service thread pid 13790 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:21:27 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:21:27 dc2oss08 kernel: : LustreError: dumping log to /tmp/lustre-log.1394004087.13790 Mar 5 02:21:28 dc2oss08 kernel: : Lustre: Service thread pid 32174 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:21:28 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:21:28 dc2oss08 kernel: : LustreError: dumping log to /tmp/lustre-log.1394004088.32174 Mar 5 02:21:29 dc2oss12 kernel: : Lustre: dc2-OST007a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:21:29 dc2oss12 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 02:21:40 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:21:40 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 02:22:02 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:22:02 dc2oss15 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:22:12 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:22:12 dc2oss06 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 02:22:15 dc2oss12 kernel: : Lustre: Service thread pid 9333 was inactive for 250.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:22:15 dc2oss12 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:22:15 dc2oss12 kernel: : LustreError: dumping log to /tmp/lustre-log.1394004135.9333 Mar 5 02:22:47 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 02:22:47 dc2oss11 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 02:22:58 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0089_UUID' is not available for connect (no target) Mar 5 02:22:58 dc2oss13 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 02:23:00 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:23:00 dc2oss10 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 02:23:31 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:23:31 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 02:23:50 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:23:50 dc2oss08 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 02:23:50 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394004230 last 1394004230 Mar 5 02:23:50 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 02:23:53 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 02:23:53 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:24:00 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:24:00 dc2mds01 kernel: : Lustre: Skipped 35 previous similar messages Mar 5 02:24:00 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394004240 last 1394004240 Mar 5 02:24:00 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 35 previous similar messages Mar 5 02:24:27 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 02:24:27 dc2oss10 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 02:24:59 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 02:24:59 dc2oss08 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 02:24:56 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 02:24:56 dc2oss16 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:25:15 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004d_UUID' is not available for connect (no target) Mar 5 02:25:15 dc2oss07 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 02:26:22 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394004382 last 1394004382 Mar 5 02:26:22 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 02:26:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 02:26:42 dc2oss09 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 02:26:53 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394004413 last 1394004413 Mar 5 02:26:53 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:26:52 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0098: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881807acd800 cur 1394004412 last 1394004412 Mar 5 02:26:52 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 02:27:01 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394004421 last 1394004421 Mar 5 02:27:01 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 02:27:12 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394004432 last 1394004432 Mar 5 02:27:12 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 02:27:32 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:27:32 dc2oss03 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 02:27:32 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394004452 last 1394004452 Mar 5 02:27:32 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 02:27:22 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 02:27:22 dc2oss05 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:27:31 dc2oss11 kernel: : Lustre: 16232:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394004451 last 1394004451 Mar 5 02:27:31 dc2oss11 kernel: : Lustre: 16232:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 02:27:27 dc2oss04 kernel: : Lustre: 16430:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1394004447 last 1394004447 Mar 5 02:27:27 dc2oss04 kernel: : Lustre: 16430:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 02:27:49 dc2oss10 kernel: : Lustre: 16518:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394004469 last 1394004469 Mar 5 02:27:49 dc2oss10 kernel: : Lustre: 16518:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 02:28:00 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394004480 last 1394004480 Mar 5 02:28:00 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 02:28:16 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:28:16 dc2oss02 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:28:16 dc2oss02 kernel: : Lustre: 15930:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394004496 last 1394004496 Mar 5 02:28:16 dc2oss02 kernel: : Lustre: 15930:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 02:28:23 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394004503 last 1394004503 Mar 5 02:28:23 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:28:44 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:28:44 dc2oss07 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 02:28:44 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1394004524 last 1394004524 Mar 5 02:28:44 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 02:28:35 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:28:35 dc2oss04 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 02:28:40 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 02:28:40 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 02:28:40 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1394004520 last 1394004520 Mar 5 02:28:40 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 02:28:45 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:28:45 dc2oss11 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 02:28:53 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:28:53 dc2oss16 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 02:28:44 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 02:28:44 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 02:29:07 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:29:07 dc2oss01 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 02:29:07 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394004547 last 1394004547 Mar 5 02:29:07 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 02:29:37 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 02:29:37 dc2oss15 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 02:30:10 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:30:10 dc2oss14 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 02:29:50 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 02:29:50 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 02:30:27 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 02:30:27 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:30:46 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 02:30:46 dc2oss12 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:31:19 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 02:31:19 dc2oss03 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 02:31:00 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:31:00 dc2oss09 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 02:31:39 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 02:31:39 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 02:31:30 dc2oss12 kernel: : Lustre: 32763:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-335), not sending early reply Mar 5 02:31:31 dc2oss12 kernel: : Lustre: 32675:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-335), not sending early reply Mar 5 02:31:35 dc2oss12 kernel: : LustreError: 32759:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 940+0s req@ffff880ef3136850 x1442912232291692/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394004695 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:31:35 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 02:31:35 dc2oss12 kernel: : Lustre: Service thread pid 32759 completed after 940.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:31:36 dc2oss12 kernel: : LustreError: 14612:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 940+0s req@ffff880abf202400 x1442912232292051/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394004696 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:31:36 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 02:31:36 dc2oss12 kernel: : Lustre: Service thread pid 14612 completed after 940.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:31:36 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:31:36 dc2oss12 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 02:31:38 dc2oss12 kernel: : Lustre: 32695:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-335), not sending early reply Mar 5 02:31:43 dc2oss12 kernel: : LustreError: 14569:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 940+0s req@ffff88070a597c00 x1442912232294711/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394004703 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:31:43 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 02:31:43 dc2oss12 kernel: : Lustre: Service thread pid 14569 completed after 940.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:31:44 dc2oss13 kernel: : Lustre: dc2-OST0085: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:31:44 dc2oss13 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 02:32:03 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:32:03 dc2oss15 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 02:32:13 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:32:13 dc2oss06 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 02:32:41 dc2oss12 kernel: : Lustre: 9254:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-276), not sending early reply Mar 5 02:32:46 dc2oss12 kernel: : LustreError: 9333:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 881+0s req@ffff880bcfab3800 x1442912232329594/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 2 to 0 dl 1394004766 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:32:46 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 02:32:46 dc2oss12 kernel: : Lustre: Service thread pid 9333 completed after 881.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:33:01 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 02:33:01 dc2oss11 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:33:16 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:33:16 dc2oss10 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 02:33:42 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:33:42 dc2oss05 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 02:33:45 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 02:33:45 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 02:34:02 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 02:34:02 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:34:06 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:34:06 dc2oss08 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 02:34:06 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394004846 last 1394004846 Mar 5 02:34:06 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 02:34:23 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:34:23 dc2mds01 kernel: : Lustre: Skipped 22 previous similar messages Mar 5 02:34:23 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394004863 last 1394004863 Mar 5 02:34:23 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 22 previous similar messages Mar 5 02:34:39 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 02:34:39 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:35:16 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 02:35:16 dc2oss07 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:35:41 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 02:35:41 dc2oss08 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:35:56 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 02:35:56 dc2oss16 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:36:29 dc2oss09 kernel: : Lustre: 16169:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394004989 last 1394004989 Mar 5 02:36:29 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394004989 last 1394004989 Mar 5 02:36:29 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:36:38 dc2oss08 kernel: : Lustre: 27731:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-511), not sending early reply Mar 5 02:36:39 dc2oss08 kernel: : Lustre: 27669:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-511), not sending early reply Mar 5 02:36:43 dc2oss08 kernel: : LustreError: 13790:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1116+0s req@ffff88071ad36400 x1442912232329951/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 4 to 0 dl 1394005003 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:36:43 dc2oss08 kernel: : Lustre: dc2-OST004b: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 02:36:43 dc2oss08 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 02:36:43 dc2oss08 kernel: : Lustre: Service thread pid 13790 completed after 1116.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:36:44 dc2oss08 kernel: : LustreError: 32174:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1116+0s req@ffff88095332cc00 x1442912232329990/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 4 to 0 dl 1394005004 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:36:44 dc2oss08 kernel: : Lustre: Service thread pid 32174 completed after 1116.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 02:36:58 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd1db0000 cur 1394005018 last 1394005018 Mar 5 02:36:58 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 02:37:11 dc2oss13 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394005031 last 1394005031 Mar 5 02:37:11 dc2oss13 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 02:37:16 dc2oss15 kernel: : Lustre: 15918:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0097: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf83cf000 cur 1394005036 last 1394005036 Mar 5 02:37:16 dc2oss15 kernel: : Lustre: 15918:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 02:37:14 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1394005034 last 1394005034 Mar 5 02:37:14 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 02:37:17 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 02:37:17 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:37:44 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:37:44 dc2oss03 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 02:37:44 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394005064 last 1394005064 Mar 5 02:37:44 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 02:37:30 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394005050 last 1394005050 Mar 5 02:37:30 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:37:38 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 02:37:38 dc2oss05 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 02:37:52 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394005072 last 1394005072 Mar 5 02:37:52 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 02:38:07 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394005087 last 1394005087 Mar 5 02:38:07 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 02:38:00 dc2oss11 kernel: : Lustre: 15512:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394005080 last 1394005080 Mar 5 02:38:00 dc2oss11 kernel: : Lustre: 15512:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:38:25 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:38:25 dc2oss02 kernel: : Lustre: Skipped 73 previous similar messages Mar 5 02:38:25 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394005105 last 1394005105 Mar 5 02:38:25 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 73 previous similar messages Mar 5 02:38:34 dc2oss05 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394005114 last 1394005114 Mar 5 02:38:34 dc2oss05 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 02:38:36 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:38:36 dc2oss04 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 02:38:40 dc2oss14 kernel: : Lustre: 15630:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394005120 last 1394005120 Mar 5 02:38:40 dc2oss14 kernel: : Lustre: 15630:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:38:50 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:38:50 dc2oss11 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 02:38:46 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 02:38:46 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:38:44 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 02:38:44 dc2oss01 kernel: : LustreError: Skipped 11 previous similar messages Mar 5 02:38:58 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:38:58 dc2oss16 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 02:39:08 dc2oss07 kernel: : Lustre: dc2-OST0047: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:39:08 dc2oss07 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 02:39:08 dc2oss07 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1394005148 last 1394005148 Mar 5 02:39:08 dc2oss07 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 75 previous similar messages Mar 5 02:39:18 dc2oss01 kernel: : Lustre: dc2-OST0005: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:39:18 dc2oss01 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 02:39:18 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0005: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37715128 exp ffff880be6c96400 cur 1394005158 last 1394005158 Mar 5 02:39:18 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:39:52 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 02:39:52 dc2oss15 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:40:13 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:40:13 dc2oss14 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 02:40:13 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 02:40:13 dc2mds01 kernel: : LustreError: 18281:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 02:41:02 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:41:02 dc2oss09 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 02:41:14 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 02:41:14 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:41:23 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 02:41:23 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 02:41:43 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 02:41:43 dc2oss03 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 02:41:40 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:41:40 dc2oss12 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 02:41:49 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 02:41:49 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 02:41:48 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:41:48 dc2oss13 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 02:42:06 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:42:06 dc2oss15 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 02:42:18 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:42:18 dc2oss06 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 02:43:01 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 02:43:01 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 02:43:25 dc2oss10 kernel: : Lustre: dc2-OST0061: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:43:25 dc2oss10 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 02:43:43 dc2oss05 kernel: : Lustre: dc2-OST002c: Client 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp) reconnecting Mar 5 02:43:43 dc2oss05 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:44:07 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:44:07 dc2oss08 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 02:44:07 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394005447 last 1394005447 Mar 5 02:44:07 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 02:44:14 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 02:44:14 dc2oss14 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 02:44:23 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 02:44:23 dc2oss13 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:44:49 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 02:44:49 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 02:44:56 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:44:56 dc2mds01 kernel: : Lustre: Skipped 27 previous similar messages Mar 5 02:44:56 dc2mds01 kernel: : Lustre: 1369:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394005496 last 1394005496 Mar 5 02:44:56 dc2mds01 kernel: : Lustre: 1369:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 27 previous similar messages Mar 5 02:46:01 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 02:46:01 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:46:16 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 02:46:16 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 02:46:34 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c00a7e400 cur 1394005594 last 1394005594 Mar 5 02:46:34 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 02:46:58 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394005618 last 1394005618 Mar 5 02:46:58 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:47:07 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0047_UUID' is not available for connect (no target) Mar 5 02:47:07 dc2oss08 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 02:47:13 dc2oss13 kernel: : Lustre: 15836:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394005633 last 1394005632 Mar 5 02:47:13 dc2oss13 kernel: : Lustre: 15836:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 02:47:20 dc2oss15 kernel: : Lustre: 15835:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0097: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf83cf000 cur 1394005640 last 1394005640 Mar 5 02:47:20 dc2oss15 kernel: : Lustre: 15835:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:47:19 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 02:47:19 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 02:47:22 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394005642 last 1394005642 Mar 5 02:47:22 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 02:47:49 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:47:49 dc2oss03 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 02:47:49 dc2oss03 kernel: : Lustre: 17519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c03192800 cur 1394005669 last 1394005669 Mar 5 02:47:49 dc2oss03 kernel: : Lustre: 17519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 02:47:31 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394005651 last 1394005651 Mar 5 02:47:31 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 02:48:00 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1394005680 last 1394005680 Mar 5 02:48:00 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 02:48:02 dc2oss11 kernel: : Lustre: 14502:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394005682 last 1394005682 Mar 5 02:48:02 dc2oss11 kernel: : Lustre: 14502:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:48:11 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 02:48:11 dc2oss05 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 02:48:19 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394005699 last 1394005699 Mar 5 02:48:19 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 02:48:30 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:48:30 dc2oss02 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 02:48:30 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394005710 last 1394005710 Mar 5 02:48:30 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 02:48:45 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394005725 last 1394005725 Mar 5 02:48:45 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 02:48:39 dc2oss04 kernel: : Lustre: dc2-OST0029: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:48:39 dc2oss04 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 02:48:47 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 02:48:47 dc2oss04 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 02:48:44 dc2oss14 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394005724 last 1394005724 Mar 5 02:48:44 dc2oss14 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 02:48:43 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232004468537, new: 232004468536 replay: 0. see LU-617. Mar 5 02:48:57 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:48:57 dc2oss11 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 02:49:14 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:49:14 dc2oss07 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 02:49:14 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1394005754 last 1394005754 Mar 5 02:49:14 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 02:49:09 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:49:09 dc2oss16 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 02:49:12 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 02:49:12 dc2oss01 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 02:49:27 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:49:27 dc2oss01 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 02:49:27 dc2oss01 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394005767 last 1394005767 Mar 5 02:49:27 dc2oss01 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 02:50:04 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 02:50:04 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:49:49 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232004634281, new: 232004634280 replay: 0. see LU-617. Mar 5 02:50:25 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:50:25 dc2oss14 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 02:50:49 dc2oss09 kernel: : Lustre: dc2-OST005e: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 02:50:49 dc2oss09 kernel: : LustreError: 14721:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bcd7da800 x1461680444609364/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005899 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:50:49 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:50:50 dc2oss09 kernel: : LustreError: 25932:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880600af7c00 x1461680444609385/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005901 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:50:50 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:05 dc2oss09 kernel: : Lustre: dc2-OST005e: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) reconnecting Mar 5 02:51:05 dc2oss09 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 02:51:22 dc2oss11 kernel: : Lustre: dc2-OST0073: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 02:51:22 dc2oss11 kernel: : LustreError: 8658:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd6914000 x1461680444609379/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005904 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:22 dc2oss11 kernel: : LustreError: 8658:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 02:51:22 dc2oss11 kernel: : Lustre: dc2-OST0073: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:22 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 02:51:22 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 02:51:23 dc2oss16 kernel: : LustreError: 9217:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806cf5a1c00 x1461680444609367/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005902 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:23 dc2oss16 kernel: : Lustre: dc2-OST00a6: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:23 dc2oss16 kernel: : LustreError: 27760:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8801c4d2ac00 x1461680444609386/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005903 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:23 dc2oss16 kernel: : Lustre: dc2-OST00a6: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:22 dc2oss07 kernel: : Lustre: dc2-OST0046: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 02:51:23 dc2oss07 kernel: : LustreError: 6238:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8805c21be400 x1461680444609372/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005904 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:23 dc2oss07 kernel: : LustreError: 6238:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 02:51:23 dc2oss07 kernel: : Lustre: dc2-OST0046: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:23 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 02:51:23 dc2oss07 kernel: : LustreError: 14480:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800b5c90c00 x1461680444609395/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005906 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:23 dc2oss07 kernel: : Lustre: dc2-OST0046: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:22 dc2oss12 kernel: : Lustre: dc2-OST0077: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 02:51:22 dc2oss12 kernel: : LustreError: 9203:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88084c41ac00 x1461680444609369/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005903 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:22 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:23 dc2oss12 kernel: : LustreError: 9334:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8808d011e800 x1461680444609373/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394005904 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:51:23 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:51:24 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 02:51:24 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:51:45 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:51:45 dc2oss12 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 02:51:57 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 02:51:57 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 02:52:11 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 02:52:11 dc2oss03 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 02:51:51 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:51:51 dc2oss13 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 02:51:53 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 02:51:53 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 12 previous similar messages Mar 5 02:52:10 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 02:52:10 dc2oss12 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 02:52:18 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:52:18 dc2oss15 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 02:52:21 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:52:21 dc2oss06 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 02:52:35 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 02:52:36 dc2oss10 kernel: : LustreError: 15970:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd6cbd000 x1461680444611158/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006008 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:52:36 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:53:02 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 02:53:02 dc2oss16 kernel: : LustreError: 9267:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88019e3e5000 x1461680444611173/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394005998 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:53:02 dc2oss16 kernel: : Lustre: dc2-OST00a6: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:53:26 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:53:26 dc2oss10 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 02:53:32 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) Mar 5 02:53:32 dc2oss11 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 02:53:42 dc2oss11 kernel: : Lustre: dc2-OST0073: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 02:53:42 dc2oss11 kernel: : LustreError: 25337:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd3e3a800 x1461680444611161/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006100 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:53:42 dc2oss11 kernel: : LustreError: 25337:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 02:53:42 dc2oss11 kernel: : Lustre: dc2-OST0073: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:53:42 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 02:53:43 dc2oss11 kernel: : LustreError: 25339:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8802f3ecf800 x1461680444611152/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006098 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:53:43 dc2oss11 kernel: : LustreError: 25339:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 02:53:43 dc2oss11 kernel: : Lustre: dc2-OST0073: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 02:53:43 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 02:53:44 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:53:44 dc2oss05 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 02:54:12 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:54:12 dc2oss08 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 02:54:12 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394006052 last 1394006052 Mar 5 02:54:12 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 02:54:07 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232005376052, new: 232005376051 replay: 0. see LU-617. Mar 5 02:54:25 dc2oss09 kernel: : Lustre: Service thread pid 25909 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 02:54:25 dc2oss09 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 02:54:25 dc2oss09 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 02:54:25 dc2oss09 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 02:54:25 dc2oss09 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 02:54:25 dc2oss09 kernel: : LustreError: dumping log to /tmp/lustre-log.1394006065.25909 Mar 5 02:54:26 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 02:54:26 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 02:54:58 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 02:54:58 dc2oss14 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 02:55:01 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:55:01 dc2mds01 kernel: : Lustre: Skipped 16 previous similar messages Mar 5 02:55:01 dc2mds01 kernel: : Lustre: 5943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394006101 last 1394006101 Mar 5 02:55:01 dc2mds01 kernel: : Lustre: 5943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages Mar 5 02:56:02 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 02:56:02 dc2oss16 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 02:56:03 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 02:56:03 dc2oss10 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 02:56:37 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394006197 last 1394006197 Mar 5 02:56:37 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 02:57:07 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394006227 last 1394006227 Mar 5 02:57:07 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 02:57:18 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1394006238 last 1394006238 Mar 5 02:57:18 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 02:57:28 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394006248 last 1394006248 Mar 5 02:57:28 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 02:57:49 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:57:49 dc2oss03 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 02:57:49 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394006269 last 1394006269 Mar 5 02:57:49 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 02:57:33 dc2oss15 kernel: : Lustre: 15946:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394006253 last 1394006253 Mar 5 02:57:33 dc2oss15 kernel: : Lustre: 15946:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 02:57:33 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394006253 last 1394006253 Mar 5 02:57:33 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 02:57:42 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 02:57:42 dc2oss07 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 02:58:05 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394006285 last 1394006285 Mar 5 02:58:05 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 02:58:02 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 02:58:02 dc2oss08 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 02:58:04 dc2oss11 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394006284 last 1394006284 Mar 5 02:58:04 dc2oss11 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 02:58:16 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 02:58:16 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:58:24 dc2oss06 kernel: : Lustre: 14485:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394006304 last 1394006304 Mar 5 02:58:24 dc2oss06 kernel: : Lustre: 14485:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 02:58:37 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:58:37 dc2oss02 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 02:58:37 dc2oss02 kernel: : Lustre: 16264:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394006317 last 1394006317 Mar 5 02:58:37 dc2oss02 kernel: : Lustre: 16264:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 02:58:50 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 02:58:50 dc2oss03 kernel: : LustreError: 21181:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8804e2a4a800 x1461679859461334/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006363 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:58:50 dc2oss03 kernel: : LustreError: 21181:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 02:58:50 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 02:58:50 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 02:58:51 dc2oss03 kernel: : LustreError: 20998:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806b83a1c00 x1461679859461359/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006377 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:58:51 dc2oss03 kernel: : LustreError: 20998:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 02:58:51 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 02:58:51 dc2oss03 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 02:58:32 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 02:58:32 dc2oss05 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 02:58:35 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 02:58:35 dc2oss08 kernel: : LustreError: 27748:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881811ece050 x1461679859461343/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006350 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:58:35 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 02:58:35 dc2oss08 kernel: : LustreError: 14646:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd370e400 x1461679859461344/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006350 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 02:58:35 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 02:58:40 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:58:40 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:58:40 dc2oss04 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 02:58:40 dc2oss04 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 02:58:46 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394006326 last 1394006326 Mar 5 02:58:46 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 02:58:49 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from 86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp t0 exp ffff8805201c8800 cur 1394006329 last 1394006329 Mar 5 02:58:49 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 02:58:57 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001d_UUID' is not available for connect (no target) Mar 5 02:58:57 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 02:59:02 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:59:02 dc2oss11 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 02:59:10 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:59:10 dc2oss16 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 02:59:19 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:59:19 dc2oss07 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 02:59:19 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394006359 last 1394006359 Mar 5 02:59:19 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 02:59:31 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 02:59:31 dc2oss01 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 02:59:31 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394006371 last 1394006371 Mar 5 02:59:31 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 02:59:34 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 02:59:34 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:00:08 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 03:00:08 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:00:28 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:00:28 dc2oss14 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 03:00:28 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 03:00:29 dc2oss04 kernel: : LustreError: 20207:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880897676c00 x1461679898274956/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006456 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:29 dc2oss04 kernel: : LustreError: 20207:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 03:00:29 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 03:00:29 dc2oss04 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 03:00:29 dc2oss04 kernel: : LustreError: 16253:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b086cfc00 x1461679898274931/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006453 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:29 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 03:00:45 dc2oss02 kernel: : Lustre: dc2-OST000c: Client 1cf2a96a-abeb-965f-c583-cd537465f0c5 (at 149.165.226.205@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 03:00:45 dc2oss02 kernel: : LustreError: 20663:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8807922fb400 x1461691363715592/t0(0) o4->1cf2a96a-abeb-965f-c583-cd537465f0c5@149.165.226.205@tcp:0/0 lens 448/416 e 1 to 0 dl 1394006478 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:45 dc2oss02 kernel: : Lustre: dc2-OST000c: Bulk IO write error with 1cf2a96a-abeb-965f-c583-cd537465f0c5 (at 149.165.226.205@tcp), client will retry: rc -110 Mar 5 03:00:46 dc2oss02 kernel: : Lustre: dc2-OST0011: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 03:00:47 dc2oss02 kernel: : LustreError: 21102:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880790dd1800 x1461679898274976/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006478 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:47 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 03:00:46 dc2oss07 kernel: : Lustre: dc2-OST0046: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 03:00:46 dc2oss07 kernel: : LustreError: 6229:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88073f5f6000 x1461680444621448/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006498 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:46 dc2oss07 kernel: : LustreError: 6229:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:00:46 dc2oss07 kernel: : Lustre: dc2-OST0046: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:00:46 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:00:48 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.2@o2ib Mar 5 03:00:48 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.2@o2ib Mar 5 03:00:48 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 03:00:48 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 03:00:48 dc2oss11 kernel: : Lustre: dc2-OST006c: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 03:00:49 dc2oss11 kernel: : LustreError: 25382:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd4d79800 x1461680444621429/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394006533 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:49 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:00:58 dc2oss12 kernel: : Lustre: dc2-OST0077: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 03:00:58 dc2oss12 kernel: : LustreError: 14585:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8807743a2000 x1461680444621440/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006495 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:00:58 dc2oss12 kernel: : LustreError: 14585:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:00:58 dc2oss12 kernel: : Lustre: dc2-OST0077: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:00:58 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:01:05 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 03:01:06 dc2oss16 kernel: : LustreError: 9243:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88095e180800 x1461680444621461/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006500 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:01:06 dc2oss16 kernel: : LustreError: 9243:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 03:01:06 dc2oss16 kernel: : Lustre: dc2-OST00a6: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:01:06 dc2oss16 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 03:01:07 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:01:07 dc2oss09 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 03:01:17 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232006579279, new: 232006579278 replay: 0. see LU-617. Mar 5 03:01:38 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 03:01:38 dc2oss06 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 03:01:49 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:01:49 dc2oss12 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 03:01:54 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:01:54 dc2oss13 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 03:02:15 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 03:02:15 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:02:03 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 03:02:03 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:01:58 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 03:01:59 dc2oss10 kernel: : LustreError: 1158:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fe4f2850 x1461680444621490/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394006583 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:01:59 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:01:58 dc2oss07 kernel: : Lustre: dc2-OST0046: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 03:01:58 dc2oss07 kernel: : LustreError: 25240:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880069f08800 x1461680444622413/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394006561 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 03:01:58 dc2oss07 kernel: : LustreError: 25240:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:01:58 dc2oss07 kernel: : Lustre: dc2-OST0046: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:01:58 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:01:53 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:01:53 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 03:01:57 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232006691629, new: 232006691628 replay: 0. see LU-617. Mar 5 03:02:24 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:02:24 dc2oss06 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 03:02:30 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:02:30 dc2oss15 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 03:03:03 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 03:03:03 dc2oss12 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:03:32 dc2oss11 kernel: : Lustre: Service thread pid 25455 was inactive for 216.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394006612.25455 Mar 5 03:03:32 dc2oss11 kernel: : Lustre: Service thread pid 25354 was inactive for 216.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 03:03:32 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 03:03:26 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:03:26 dc2oss10 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 03:03:45 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:03:45 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 03:03:53 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 03:03:53 dc2oss11 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:04:39 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:04:39 dc2oss08 kernel: : Lustre: Skipped 259 previous similar messages Mar 5 03:04:39 dc2oss08 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394006679 last 1394006679 Mar 5 03:04:39 dc2oss08 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 260 previous similar messages Mar 5 03:04:48 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 03:04:48 dc2oss13 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 03:05:07 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 03:05:07 dc2oss14 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:05:02 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 03:05:02 dc2mds01 kernel: : Lustre: Skipped 30 previous similar messages Mar 5 03:05:02 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394006702 last 1394006702 Mar 5 03:05:02 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 30 previous similar messages Mar 5 03:05:36 dc2oss09 kernel: : Lustre: 22664:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply Mar 5 03:05:41 dc2oss09 kernel: : LustreError: 25909:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 876+0s req@ffff8818098b3850 x1461680444610087/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 4 to 0 dl 1394006741 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 03:05:41 dc2oss09 kernel: : LustreError: 25909:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:05:41 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:05:41 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:05:41 dc2oss09 kernel: : Lustre: Service thread pid 25909 completed after 876.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 03:06:03 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 03:06:03 dc2oss10 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:06:03 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 03:06:03 dc2oss16 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:06:25 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232007451888, new: 232007451887 replay: 0. see LU-617. Mar 5 03:06:39 dc2oss09 kernel: : Lustre: 16367:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394006799 last 1394006799 Mar 5 03:06:39 dc2oss09 kernel: : Lustre: 16367:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 03:07:08 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394006828 last 1394006828 Mar 5 03:07:08 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 03:07:27 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1394006847 last 1394006847 Mar 5 03:07:27 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 03:07:34 dc2oss12 kernel: : Lustre: 16043:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394006854 last 1394006854 Mar 5 03:07:34 dc2oss12 kernel: : Lustre: 16043:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 03:07:52 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:07:52 dc2oss03 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 03:07:52 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1394006872 last 1394006872 Mar 5 03:07:52 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 03:07:33 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394006853 last 1394006853 Mar 5 03:07:33 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 03:07:52 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394006872 last 1394006872 Mar 5 03:07:52 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 03:08:05 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394006885 last 1394006885 Mar 5 03:08:05 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 03:08:07 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394006887 last 1394006887 Mar 5 03:08:07 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:08:14 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 03:08:14 dc2oss08 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 03:08:26 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394006906 last 1394006906 Mar 5 03:08:26 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 03:08:41 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:08:41 dc2oss02 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 03:08:41 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394006921 last 1394006921 Mar 5 03:08:41 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 03:08:39 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 03:08:39 dc2oss05 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:08:49 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394006929 last 1394006929 Mar 5 03:08:49 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 03:08:50 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394006930 last 1394006930 Mar 5 03:08:50 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 03:08:45 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:08:45 dc2oss04 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 03:09:09 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:09:09 dc2oss11 kernel: : Lustre: Skipped 137 previous similar messages Mar 5 03:09:11 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:09:11 dc2oss16 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 03:09:12 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 03:09:12 dc2oss04 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 03:09:20 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:09:20 dc2oss07 kernel: : Lustre: Skipped 140 previous similar messages Mar 5 03:09:20 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394006960 last 1394006960 Mar 5 03:09:20 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 137 previous similar messages Mar 5 03:09:35 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 03:09:35 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:09:32 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:09:32 dc2oss01 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 03:09:32 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394006972 last 1394006972 Mar 5 03:09:32 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 03:10:06 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 03:10:06 dc2oss07 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:09:57 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 03:09:57 dc2oss01 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 03:10:11 dc2oss11 kernel: : Lustre: 8633:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-15), not sending early reply Mar 5 03:10:16 dc2oss11 kernel: : LustreError: 8739:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 620+0s req@ffff8801c22e5400 x1461680444621463/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 2 to 0 dl 1394007016 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:10:16 dc2oss11 kernel: : Lustre: dc2-OST0073: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 03:10:16 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:10:16 dc2oss11 kernel: : Lustre: Service thread pid 8739 completed after 620.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 03:10:16 dc2oss11 kernel: : LustreError: 25354:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 620+0s req@ffff8814d17d5850 x1461680444621464/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 2 to 0 dl 1394007016 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:10:16 dc2oss11 kernel: : Lustre: Service thread pid 25455 completed after 620.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 03:10:16 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:10:39 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:10:39 dc2oss14 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 03:10:55 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232008227944, new: 232008227943 replay: 0. see LU-617. Mar 5 03:11:21 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a4_UUID' is not available for connect (no target) Mar 5 03:11:21 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 03:11:14 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:11:14 dc2oss09 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 03:11:44 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 03:11:44 dc2oss06 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:11:53 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:11:53 dc2oss12 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 03:12:20 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 03:12:20 dc2oss03 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:12:08 dc2oss13 kernel: : Lustre: dc2-OST007f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:12:08 dc2oss13 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 03:12:29 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:12:29 dc2oss06 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 03:12:36 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:12:36 dc2oss15 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 03:12:50 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 03:12:50 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 03:12:43 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:12:43 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 03:13:07 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 03:13:07 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 03:13:42 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:13:42 dc2oss10 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 03:13:45 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:13:45 dc2oss05 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 03:14:37 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 03:14:37 dc2oss11 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 03:14:45 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:14:45 dc2oss08 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 03:14:45 dc2oss08 kernel: : Lustre: 16028:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394007285 last 1394007285 Mar 5 03:14:45 dc2oss08 kernel: : Lustre: 16028:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 03:14:52 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 03:14:52 dc2oss13 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:15:07 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 03:15:07 dc2mds01 kernel: : Lustre: Skipped 32 previous similar messages Mar 5 03:15:07 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394007307 last 1394007307 Mar 5 03:15:07 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 32 previous similar messages Mar 5 03:15:31 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 03:15:31 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:16:09 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 03:16:09 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:16:11 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 03:16:11 dc2oss10 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 03:16:40 dc2oss09 kernel: : Lustre: 16391:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1394007400 last 1394007400 Mar 5 03:16:40 dc2oss09 kernel: : Lustre: 16391:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 03:17:17 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394007437 last 1394007437 Mar 5 03:17:17 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 03:17:13 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x341:0x0]] [valid 68719476736] Mar 5 03:17:13 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 03:17:35 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394007455 last 1394007455 Mar 5 03:17:35 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 03:17:57 dc2oss03 kernel: : Lustre: dc2-OST001a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:17:57 dc2oss03 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 03:17:57 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c03192800 cur 1394007477 last 1394007477 Mar 5 03:17:57 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 03:17:38 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394007458 last 1394007458 Mar 5 03:17:38 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 03:17:41 dc2oss04 kernel: : Lustre: 16359:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394007461 last 1394007461 Mar 5 03:17:41 dc2oss04 kernel: : Lustre: 16359:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 03:17:56 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394007476 last 1394007476 Mar 5 03:17:56 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 03:18:09 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394007489 last 1394007489 Mar 5 03:18:09 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 03:18:11 dc2oss11 kernel: : Lustre: 15357:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394007491 last 1394007491 Mar 5 03:18:11 dc2oss11 kernel: : Lustre: 15357:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 03:18:25 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 03:18:25 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 03:18:28 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394007508 last 1394007508 Mar 5 03:18:28 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 03:18:43 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:18:43 dc2oss02 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 03:18:43 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394007523 last 1394007523 Mar 5 03:18:43 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 03:18:45 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0039_UUID' is not available for connect (no target) Mar 5 03:18:45 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:18:50 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394007530 last 1394007530 Mar 5 03:18:50 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 03:18:48 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:18:48 dc2oss04 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 03:18:55 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394007535 last 1394007535 Mar 5 03:18:55 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 03:19:14 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:19:14 dc2oss16 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 03:19:10 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:19:10 dc2oss11 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 03:19:20 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:19:20 dc2oss07 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 03:19:20 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394007560 last 1394007560 Mar 5 03:19:20 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 03:19:16 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 03:19:16 dc2oss04 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:19:51 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:19:51 dc2oss01 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 03:19:51 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394007591 last 1394007591 Mar 5 03:19:51 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 03:19:58 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 03:19:58 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 03:20:14 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 03:20:14 dc2oss07 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 03:20:44 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 03:20:44 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 03:20:45 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:20:45 dc2oss14 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 03:21:27 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:21:27 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 03:21:28 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 03:21:28 dc2oss15 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 03:21:49 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 03:21:49 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 03:22:04 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:22:04 dc2oss12 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 03:22:15 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:22:15 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 03:22:35 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 03:22:35 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:22:13 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232010137525, new: 232010137524 replay: 0. see LU-617. Mar 5 03:22:37 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:22:37 dc2oss15 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 03:22:40 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:22:40 dc2oss06 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 03:22:44 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:22:44 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 35 previous similar messages Mar 5 03:23:14 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 03:23:14 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 03:23:14 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 03:23:14 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:23:45 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:23:45 dc2oss10 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 03:23:49 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:23:49 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 03:24:42 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 03:24:42 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:25:04 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:25:04 dc2oss08 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 03:25:04 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394007904 last 1394007904 Mar 5 03:25:04 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 03:25:12 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 03:25:12 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 03:25:15 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:25:15 dc2mds01 kernel: : Lustre: Skipped 38 previous similar messages Mar 5 03:25:15 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394007915 last 1394007915 Mar 5 03:25:15 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 38 previous similar messages Mar 5 03:25:34 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 03:25:34 dc2oss14 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:26:12 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 03:26:12 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 03:26:40 dc2oss09 kernel: : Lustre: 16181:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394008000 last 1394008000 Mar 5 03:26:40 dc2oss09 kernel: : Lustre: 16181:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 03:27:20 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005d_UUID' is not available for connect (no target) Mar 5 03:27:20 dc2oss10 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:27:13 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232011002563, new: 232011002562 replay: 0. see LU-617. Mar 5 03:27:28 dc2oss16 kernel: : Lustre: 17124:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394008048 last 1394008048 Mar 5 03:27:28 dc2oss16 kernel: : Lustre: 17124:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 03:27:41 dc2oss12 kernel: : Lustre: 16128:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394008061 last 1394008061 Mar 5 03:27:41 dc2oss12 kernel: : Lustre: 16128:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 03:28:01 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:28:01 dc2oss03 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 03:28:01 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0df95000 cur 1394008081 last 1394008081 Mar 5 03:28:01 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 03:27:44 dc2oss13 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394008064 last 1394008064 Mar 5 03:27:44 dc2oss13 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 03:27:50 dc2oss04 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394008070 last 1394008070 Mar 5 03:27:50 dc2oss04 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 03:27:58 dc2oss15 kernel: : Lustre: 15672:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394008078 last 1394008078 Mar 5 03:27:58 dc2oss15 kernel: : Lustre: 15672:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 03:28:20 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394008100 last 1394008100 Mar 5 03:28:20 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 03:28:16 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394008096 last 1394008096 Mar 5 03:28:16 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 03:28:34 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394008114 last 1394008114 Mar 5 03:28:34 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 03:28:48 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:28:48 dc2oss02 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 03:28:48 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394008128 last 1394008128 Mar 5 03:28:48 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 03:28:50 dc2oss04 kernel: : Lustre: dc2-OST0022: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:28:50 dc2oss04 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 03:28:57 dc2oss05 kernel: : Lustre: 14614:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0030: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be2951800 cur 1394008137 last 1394008137 Mar 5 03:28:57 dc2oss05 kernel: : Lustre: 14614:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 03:28:57 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394008137 last 1394008137 Mar 5 03:28:57 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 03:29:07 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 03:29:07 dc2oss05 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 03:29:12 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:29:12 dc2oss11 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 03:29:13 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 03:29:13 dc2oss08 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 03:29:15 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:29:15 dc2oss16 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 03:29:12 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232011334919, new: 232011334918 replay: 0. see LU-617. Mar 5 03:29:27 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:29:27 dc2oss07 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 03:29:27 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394008167 last 1394008167 Mar 5 03:29:27 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 03:29:37 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 03:29:37 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 03:30:01 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:30:01 dc2oss01 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 03:30:01 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394008201 last 1394008201 Mar 5 03:30:01 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 03:30:33 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 03:30:33 dc2oss01 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:30:47 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:30:47 dc2oss14 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 03:30:50 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 03:30:50 dc2oss09 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 03:31:12 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004d_UUID' is not available for connect (no target) Mar 5 03:31:12 dc2oss07 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:31:34 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:31:34 dc2oss09 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 03:31:37 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 03:31:37 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 03:31:36 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232011764533, new: 232011764532 replay: 0. see LU-617. Mar 5 03:32:05 dc2oss12 kernel: : Lustre: dc2-OST0075: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) reconnecting Mar 5 03:32:05 dc2oss12 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 03:32:36 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0024_UUID' is not available for connect (no target) Mar 5 03:32:36 dc2oss03 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 03:32:26 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:32:26 dc2oss13 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 03:32:38 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:32:38 dc2oss15 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 03:32:49 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 03:32:49 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:32:53 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:32:53 dc2oss06 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 03:33:23 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 03:33:23 dc2oss12 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 03:33:48 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:33:48 dc2oss10 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 03:34:03 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:34:03 dc2oss05 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 03:33:58 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:33:58 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 03:34:28 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 03:34:28 dc2oss02 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 03:35:05 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:35:05 dc2oss08 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 03:35:05 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394008505 last 1394008505 Mar 5 03:35:05 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 03:35:10 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 03:35:10 dc2oss11 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 03:35:35 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 03:35:35 dc2oss13 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 03:36:15 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 03:36:15 dc2oss14 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 03:36:15 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:36:15 dc2mds01 kernel: : Lustre: Skipped 29 previous similar messages Mar 5 03:36:15 dc2mds01 kernel: : Lustre: 5939:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394008575 last 1394008575 Mar 5 03:36:15 dc2mds01 kernel: : Lustre: 5939:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 29 previous similar messages Mar 5 03:36:47 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394008607 last 1394008607 Mar 5 03:36:47 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 03:37:09 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009a_UUID' is not available for connect (no target) Mar 5 03:37:09 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 03:37:29 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394008649 last 1394008649 Mar 5 03:37:29 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 03:37:42 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394008662 last 1394008662 Mar 5 03:37:42 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 03:37:44 dc2oss13 kernel: : Lustre: 14447:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394008664 last 1394008664 Mar 5 03:37:44 dc2oss13 kernel: : Lustre: 14447:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:38:06 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:38:06 dc2oss03 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 03:38:06 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394008686 last 1394008686 Mar 5 03:38:06 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 03:37:50 dc2oss04 kernel: : Lustre: 16563:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394008670 last 1394008670 Mar 5 03:37:50 dc2oss04 kernel: : Lustre: 16563:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 03:38:03 dc2oss15 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0098: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881807acd800 cur 1394008683 last 1394008683 Mar 5 03:38:03 dc2oss15 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 03:38:08 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 03:38:08 dc2oss07 kernel: : LustreError: 6379:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88009f7e0000 x1461680324050883/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008729 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:08 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:09 dc2oss07 kernel: : LustreError: 25267:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf86ea850 x1461680324050853/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008727 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:09 dc2oss07 kernel: : LustreError: 25267:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:38:09 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:09 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:38:22 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from 025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp t0 exp ffff881157975400 cur 1394008702 last 1394008702 Mar 5 03:38:22 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 03:38:22 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 03:38:22 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:38:17 dc2oss05 kernel: : Lustre: dc2-OST002f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 03:38:17 dc2oss05 kernel: : LustreError: 29967:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880a89518c00 x1461680324050823/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008736 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:17 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:18 dc2oss05 kernel: : LustreError: 29976:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880796e30800 x1461680324050817/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008736 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:18 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:15 dc2oss16 kernel: : Lustre: dc2-OST009f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 03:38:15 dc2oss16 kernel: : LustreError: 9275:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8805b3923400 x1461680324050834/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008727 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:15 dc2oss16 kernel: : LustreError: 9275:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 03:38:15 dc2oss16 kernel: : Lustre: dc2-OST009f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:15 dc2oss16 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 03:38:16 dc2oss16 kernel: : LustreError: 9224:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803dda14800 x1461680324050880/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008735 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:16 dc2oss16 kernel: : LustreError: 9224:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 03:38:16 dc2oss16 kernel: : Lustre: dc2-OST009f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:16 dc2oss16 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 03:38:22 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394008702 last 1394008702 Mar 5 03:38:22 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 03:38:34 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394008714 last 1394008714 Mar 5 03:38:34 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 03:38:26 dc2oss08 kernel: : Lustre: dc2-OST004d: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 03:38:27 dc2oss08 kernel: : LustreError: 32212:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b78186800 x1461680324050891/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008742 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:27 dc2oss08 kernel: : LustreError: 32212:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 03:38:27 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:27 dc2oss08 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 03:38:27 dc2oss08 kernel: : LustreError: 14577:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881806832850 x1461680324050852/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394008739 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 03:38:27 dc2oss08 kernel: : LustreError: 14577:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 03:38:27 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 03:38:27 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 03:38:49 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:38:49 dc2oss02 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 03:38:49 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1394008729 last 1394008729 Mar 5 03:38:49 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 03:39:02 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394008742 last 1394008742 Mar 5 03:39:02 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 03:39:12 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:39:12 dc2oss11 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 03:39:09 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:39:09 dc2oss04 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 03:39:16 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394008756 last 1394008756 Mar 5 03:39:16 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 03:39:23 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0047_UUID' is not available for connect (no target) Mar 5 03:39:23 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:39:28 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:39:28 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 03:39:30 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:39:30 dc2oss07 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 03:39:30 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394008770 last 1394008770 Mar 5 03:39:30 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 03:39:33 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 03:39:33 dc2oss05 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 03:39:44 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 03:39:44 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:40:15 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:40:15 dc2oss01 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 03:40:15 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394008815 last 1394008815 Mar 5 03:40:15 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 03:40:52 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:40:52 dc2oss14 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 03:40:59 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0062_UUID' is not available for connect (no target) Mar 5 03:40:59 dc2oss09 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:41:12 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 03:41:12 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 03:41:16 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 03:41:16 dc2oss01 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 03:41:40 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:41:40 dc2oss09 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 03:42:07 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:42:07 dc2oss12 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 03:42:27 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:42:27 dc2oss13 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 03:42:48 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 03:42:48 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:42:40 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:42:40 dc2oss15 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 03:42:40 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 03:42:40 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 03:42:51 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 03:42:51 dc2oss06 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 03:42:56 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:42:56 dc2oss06 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 03:43:32 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 03:43:32 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:43:49 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:43:49 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 03:44:05 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:44:05 dc2oss05 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 03:44:28 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 03:44:28 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 03:45:09 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:45:09 dc2oss08 kernel: : Lustre: Skipped 69 previous similar messages Mar 5 03:45:09 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394009109 last 1394009108 Mar 5 03:45:09 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 67 previous similar messages Mar 5 03:45:13 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 03:45:13 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 03:45:13 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:45:13 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) Skipped 4 previous similar messages Mar 5 03:45:45 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008b_UUID' is not available for connect (no target) Mar 5 03:45:45 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:46:33 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 03:46:33 dc2oss14 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:46:52 dc2oss09 kernel: : Lustre: 16266:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1394009212 last 1394009212 Mar 5 03:46:52 dc2oss09 kernel: : Lustre: 16266:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:46:53 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 03:46:53 dc2mds01 kernel: : Lustre: Skipped 13 previous similar messages Mar 5 03:46:53 dc2mds01 kernel: : Lustre: 9759:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394009213 last 1394009213 Mar 5 03:46:53 dc2mds01 kernel: : Lustre: 9759:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 13 previous similar messages Mar 5 03:47:31 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394009251 last 1394009251 Mar 5 03:47:31 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 03:47:33 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 03:47:33 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:47:43 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394009263 last 1394009263 Mar 5 03:47:43 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:47:47 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394009267 last 1394009267 Mar 5 03:47:47 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 03:48:18 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:48:18 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 03:48:18 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394009298 last 1394009298 Mar 5 03:48:18 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 03:47:52 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394009272 last 1394009272 Mar 5 03:47:52 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 03:48:13 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394009293 last 1394009293 Mar 5 03:48:13 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 03:48:24 dc2oss10 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394009304 last 1394009304 Mar 5 03:48:24 dc2oss10 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:48:30 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 03:48:30 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 03:48:24 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394009304 last 1394009304 Mar 5 03:48:24 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 03:48:34 dc2oss06 kernel: : Lustre: 15955:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394009314 last 1394009314 Mar 5 03:48:34 dc2oss06 kernel: : Lustre: 15955:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 03:48:51 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:48:51 dc2oss02 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 03:48:51 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394009331 last 1394009331 Mar 5 03:48:51 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 03:49:07 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394009347 last 1394009347 Mar 5 03:49:07 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 03:49:09 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232014762043, new: 232014762042 replay: 0. see LU-617. Mar 5 03:49:13 dc2oss11 kernel: : Lustre: dc2-OST0069: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:49:13 dc2oss11 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 03:49:17 dc2oss05 kernel: : Lustre: 15845:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394009357 last 1394009357 Mar 5 03:49:17 dc2oss05 kernel: : Lustre: 15845:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 03:49:15 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:49:15 dc2oss04 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 03:49:23 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 03:49:23 dc2oss08 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 03:49:31 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:49:31 dc2oss16 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 03:49:33 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:49:33 dc2oss07 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 03:49:33 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394009373 last 1394009373 Mar 5 03:49:33 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 03:49:36 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 03:49:36 dc2oss05 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 03:49:50 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0019_UUID' is not available for connect (no target) Mar 5 03:49:50 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:50:18 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:50:18 dc2oss01 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 03:50:18 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394009418 last 1394009418 Mar 5 03:50:18 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 03:50:55 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:50:55 dc2oss14 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 03:51:14 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 03:51:14 dc2oss07 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:51:45 dc2oss09 kernel: : Lustre: dc2-OST0057: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:51:45 dc2oss09 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 03:51:47 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 03:51:47 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 03:51:54 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 03:51:54 dc2oss01 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 03:52:24 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:52:24 dc2oss12 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 03:52:30 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:52:30 dc2oss13 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 03:52:44 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:52:44 dc2oss15 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 03:53:12 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 03:53:12 dc2oss03 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 03:52:52 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002c_UUID' is not available for connect (no target) Mar 5 03:52:52 dc2oss06 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 03:53:01 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a4_UUID' is not available for connect (no target) Mar 5 03:53:01 dc2oss15 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 03:53:07 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:53:07 dc2oss06 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 03:53:33 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 03:53:33 dc2oss12 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:53:53 dc2oss10 kernel: : Lustre: dc2-OST0067: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:53:53 dc2oss10 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 03:54:08 dc2oss05 kernel: : Lustre: dc2-OST0033: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:54:08 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 03:55:09 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 03:55:09 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 03:55:16 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 03:55:16 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 03:55:22 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:55:22 dc2oss08 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 03:55:22 dc2oss08 kernel: : Lustre: 16199:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394009722 last 1394009722 Mar 5 03:55:22 dc2oss08 kernel: : Lustre: 16199:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 03:56:00 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 03:56:00 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 03:56:23 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 03:56:23 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 23 previous similar messages Mar 5 03:56:39 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0087_UUID' is not available for connect (no target) Mar 5 03:56:39 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:57:04 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1394009824 last 1394009824 Mar 5 03:57:04 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 03:56:54 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:56:54 dc2mds01 kernel: : Lustre: Skipped 34 previous similar messages Mar 5 03:56:54 dc2mds01 kernel: : Lustre: 11148:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394009814 last 1394009814 Mar 5 03:56:54 dc2mds01 kernel: : Lustre: 11148:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 34 previous similar messages Mar 5 03:57:10 dc2mds01 kernel: : Lustre: 1376:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021251e5:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 03:57:38 dc2mds01 kernel: : LustreError: 11148:0:(osd_handler.c:308:osd_iget()) no inode Mar 5 03:57:38 dc2mds01 kernel: : LustreError: 11148:0:(osd_handler.c:308:osd_iget()) Skipped 5 previous similar messages Mar 5 03:57:46 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394009866 last 1394009866 Mar 5 03:57:46 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 03:57:52 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394009872 last 1394009872 Mar 5 03:57:52 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 03:57:52 dc2oss12 kernel: : Lustre: 16126:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394009872 last 1394009872 Mar 5 03:57:52 dc2oss12 kernel: : Lustre: 16126:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 03:57:53 dc2oss04 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394009873 last 1394009873 Mar 5 03:57:53 dc2oss04 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 03:58:00 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 03:58:00 dc2oss16 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 03:58:21 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:58:21 dc2oss03 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 03:58:21 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394009901 last 1394009901 Mar 5 03:58:21 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 03:58:22 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394009902 last 1394009902 Mar 5 03:58:22 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 03:58:31 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394009911 last 1394009911 Mar 5 03:58:31 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 03:58:26 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394009906 last 1394009906 Mar 5 03:58:26 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 03:58:52 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:58:52 dc2oss02 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 03:58:52 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394009932 last 1394009932 Mar 5 03:58:52 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 03:58:38 dc2oss06 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394009918 last 1394009918 Mar 5 03:58:38 dc2oss06 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 03:59:01 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 03:59:01 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 03:59:08 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394009948 last 1394009948 Mar 5 03:59:08 dc2oss14 kernel: : Lustre: 14481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 03:59:24 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:59:24 dc2oss11 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 03:59:19 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:59:19 dc2oss04 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 03:59:35 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:59:35 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 03:59:26 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394009966 last 1394009966 Mar 5 03:59:26 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 03:59:27 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 03:59:27 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 03:59:41 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 03:59:41 dc2oss07 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 03:59:41 dc2oss07 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394009981 last 1394009981 Mar 5 03:59:41 dc2oss07 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 04:00:16 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 04:00:16 dc2oss05 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 04:00:22 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:00:22 dc2oss01 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 04:00:22 dc2oss01 kernel: : Lustre: 15329:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394010022 last 1394010022 Mar 5 04:00:22 dc2oss01 kernel: : Lustre: 15329:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 04:00:30 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 04:00:30 dc2oss04 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:00:58 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:00:58 dc2oss14 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:01:45 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:01:45 dc2oss09 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 04:01:49 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 04:01:49 dc2oss09 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 04:01:54 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 04:01:54 dc2oss07 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 04:02:07 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 04:02:07 dc2oss01 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 04:02:30 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:02:30 dc2oss13 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 04:02:31 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:02:31 dc2oss12 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 04:02:50 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:02:50 dc2oss15 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 04:02:58 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 04:02:58 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:03:35 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 04:03:35 dc2oss03 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:03:17 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:03:17 dc2oss06 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 04:03:48 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 04:03:48 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:03:42 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232017296675, new: 232017296674 replay: 0. see LU-617. Mar 5 04:03:57 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:03:57 dc2oss10 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 04:04:12 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:04:12 dc2oss05 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 04:05:07 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 04:05:07 dc2oss12 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:05:28 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 04:05:28 dc2oss02 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:05:30 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:05:30 dc2oss08 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 04:05:30 dc2oss08 kernel: : Lustre: 16068:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394010330 last 1394010330 Mar 5 04:05:30 dc2oss08 kernel: : Lustre: 16068:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 04:06:11 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 04:06:11 dc2oss13 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 04:06:14 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 04:06:14 dc2oss11 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 04:06:39 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0080_UUID' is not available for connect (no target) Mar 5 04:06:39 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 04:06:28 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 04:06:28 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 25 previous similar messages Mar 5 04:07:04 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394010424 last 1394010424 Mar 5 04:07:04 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:07:18 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 04:07:18 dc2mds01 kernel: : Lustre: Skipped 34 previous similar messages Mar 5 04:07:18 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394010438 last 1394010438 Mar 5 04:07:18 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 34 previous similar messages Mar 5 04:07:50 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05c45400 cur 1394010470 last 1394010470 Mar 5 04:07:50 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 04:07:52 dc2oss13 kernel: : Lustre: 15838:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394010472 last 1394010472 Mar 5 04:07:52 dc2oss13 kernel: : Lustre: 15838:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 04:07:47 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x35f:0x0]] [valid 68719476736] Mar 5 04:07:47 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:07:58 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394010478 last 1394010478 Mar 5 04:07:58 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 04:08:01 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394010481 last 1394010481 Mar 5 04:08:01 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 04:08:23 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:08:23 dc2oss03 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:08:23 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394010503 last 1394010503 Mar 5 04:08:23 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 04:08:26 dc2oss15 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0098: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881807acd800 cur 1394010506 last 1394010506 Mar 5 04:08:26 dc2oss15 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 04:08:36 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 04:08:36 dc2oss16 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:08:32 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394010512 last 1394010512 Mar 5 04:08:32 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 04:08:52 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:08:52 dc2oss02 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 04:08:52 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394010532 last 1394010532 Mar 5 04:08:52 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 04:08:44 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394010524 last 1394010524 Mar 5 04:08:44 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 50 previous similar messages Mar 5 04:08:49 dc2oss06 kernel: : Lustre: 16092:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394010529 last 1394010529 Mar 5 04:08:49 dc2oss06 kernel: : Lustre: 16092:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 04:09:11 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394010551 last 1394010551 Mar 5 04:09:11 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:09:28 dc2oss11 kernel: : Lustre: dc2-OST0072: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:09:28 dc2oss11 kernel: : Lustre: Skipped 59 previous similar messages Mar 5 04:09:27 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:09:27 dc2oss04 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 04:09:32 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 04:09:32 dc2oss10 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:09:33 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 04:09:33 dc2oss08 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 04:09:34 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394010574 last 1394010574 Mar 5 04:09:34 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 04:09:51 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:09:51 dc2oss16 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 04:10:00 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:10:00 dc2oss07 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 04:10:00 dc2oss07 kernel: : Lustre: 14342:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394010600 last 1394010600 Mar 5 04:10:00 dc2oss07 kernel: : Lustre: 14342:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 04:10:22 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0039_UUID' is not available for connect (no target) Mar 5 04:10:22 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 04:10:33 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:10:33 dc2oss01 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 04:10:33 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394010633 last 1394010633 Mar 5 04:10:33 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 04:10:42 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 04:10:42 dc2oss04 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:11:07 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:11:07 dc2oss14 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 04:11:50 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 04:11:50 dc2oss09 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 04:11:57 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:11:57 dc2oss09 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 04:11:54 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232018747652, new: 232018747651 replay: 0. see LU-617. Mar 5 04:12:22 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 04:12:22 dc2oss01 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 04:12:31 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:12:31 dc2oss12 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 04:12:40 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:12:40 dc2oss13 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 04:12:59 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:12:59 dc2oss15 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 04:12:51 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232018914087, new: 232018914086 replay: 0. see LU-617. Mar 5 04:13:11 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 04:13:11 dc2oss06 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 04:13:23 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:13:23 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:13:57 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 04:13:57 dc2oss03 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 04:14:05 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:14:05 dc2oss10 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 04:14:18 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:14:18 dc2oss05 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:14:20 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a4_UUID' is not available for connect (no target) Mar 5 04:14:20 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:14:19 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004d_UUID' is not available for connect (no target) Mar 5 04:14:19 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:15:28 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 04:15:28 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:15:09 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232019251133, new: 232019251132 replay: 0. see LU-617. Mar 5 04:15:49 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 04:15:49 dc2oss12 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:15:59 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:15:59 dc2oss08 kernel: : Lustre: Skipped 45 previous similar messages Mar 5 04:15:59 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394010959 last 1394010959 Mar 5 04:15:59 dc2oss08 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 45 previous similar messages Mar 5 04:16:24 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 04:16:24 dc2oss13 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:17:10 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25e45c00 cur 1394011030 last 1394011030 Mar 5 04:17:10 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 04:17:07 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 04:17:07 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 04:17:18 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 04:17:18 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:17:19 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:17:19 dc2mds01 kernel: : Lustre: Skipped 27 previous similar messages Mar 5 04:17:19 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394011039 last 1394011039 Mar 5 04:17:19 dc2mds01 kernel: : Lustre: 1373:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 27 previous similar messages Mar 5 04:17:55 dc2oss13 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1394011075 last 1394011075 Mar 5 04:17:55 dc2oss13 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 04:17:57 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394011077 last 1394011077 Mar 5 04:17:57 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 04:18:05 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394011085 last 1394011085 Mar 5 04:18:05 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 04:18:02 dc2oss12 kernel: : Lustre: 15981:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394011082 last 1394011082 Mar 5 04:18:02 dc2oss12 kernel: : Lustre: 15981:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:18:31 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:18:31 dc2oss03 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 04:18:31 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394011111 last 1394011111 Mar 5 04:18:31 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 04:18:37 dc2oss15 kernel: : Lustre: 15861:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394011117 last 1394011117 Mar 5 04:18:37 dc2oss15 kernel: : Lustre: 15861:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 04:18:45 dc2oss10 kernel: : Lustre: 15827:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394011125 last 1394011125 Mar 5 04:18:45 dc2oss10 kernel: : Lustre: 15827:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 129 previous similar messages Mar 5 04:18:50 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 04:18:50 dc2oss16 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 04:19:03 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:19:03 dc2oss02 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 04:19:03 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394011143 last 1394011143 Mar 5 04:19:03 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 04:18:56 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394011136 last 1394011136 Mar 5 04:18:56 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 04:19:11 dc2oss11 kernel: : Lustre: 16223:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394011151 last 1394011151 Mar 5 04:19:11 dc2oss11 kernel: : Lustre: 16223:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 04:19:13 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394011153 last 1394011153 Mar 5 04:19:13 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 04:19:22 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232019938002, new: 232019938001 replay: 0. see LU-617. Mar 5 04:19:30 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:19:30 dc2oss04 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 04:19:35 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 04:19:35 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 04:19:35 dc2oss05 kernel: : Lustre: 16122:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394011175 last 1394011175 Mar 5 04:19:35 dc2oss05 kernel: : Lustre: 16122:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 04:19:38 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:19:38 dc2oss11 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 04:19:49 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 04:19:49 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 04:20:01 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:20:01 dc2oss16 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 04:20:01 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:20:01 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:20:01 dc2oss07 kernel: : Lustre: Skipped 142 previous similar messages Mar 5 04:20:01 dc2oss07 kernel: : Lustre: Skipped 142 previous similar messages Mar 5 04:20:01 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394011201 last 1394011201 Mar 5 04:20:01 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 142 previous similar messages Mar 5 04:20:39 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 04:20:39 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 04:20:44 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:20:44 dc2oss01 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 04:20:44 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394011244 last 1394011244 Mar 5 04:20:44 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 04:20:53 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0019_UUID' is not available for connect (no target) Mar 5 04:20:53 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:21:15 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 04:21:15 dc2oss05 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 04:21:15 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:21:15 dc2oss14 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 04:21:10 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x377:0x0]] [valid 68719476736] Mar 5 04:21:10 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:21:10 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x378:0x0]] [valid 68719476736] Mar 5 04:21:36 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x383:0x0]] [valid 68719476736] Mar 5 04:22:05 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:22:05 dc2oss09 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 04:22:29 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 04:22:29 dc2oss09 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 04:22:15 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232020372508, new: 232020372507 replay: 0. see LU-617. Mar 5 04:22:33 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:22:33 dc2oss12 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 04:22:46 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:22:46 dc2oss13 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 04:22:59 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:22:59 dc2oss15 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 04:23:16 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 04:23:16 dc2oss06 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 04:23:23 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:23:23 dc2oss06 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:24:00 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 04:24:00 dc2oss03 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 04:24:22 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:24:22 dc2oss05 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 04:24:17 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:24:17 dc2oss10 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 04:24:16 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 04:24:16 dc2oss01 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:24:27 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 04:24:27 dc2oss15 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 04:24:30 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 04:24:30 dc2oss07 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 04:25:47 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 04:25:47 dc2oss02 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:25:59 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:25:59 dc2oss08 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 04:25:59 dc2oss08 kernel: : Lustre: 16172:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394011559 last 1394011559 Mar 5 04:25:59 dc2oss08 kernel: : Lustre: 16172:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 04:26:39 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 04:26:39 dc2oss12 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:26:39 dc2oss14 kernel: : Lustre: dc2-OST008c: Client 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 04:26:40 dc2oss14 kernel: : LustreError: 22633:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be1984800 x1461680408013579/t0(0) o3->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/400 e 1 to 0 dl 1394011646 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:26:40 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 04:26:40 dc2oss14 kernel: : LustreError: 22626:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b59dd8000 x1461680408013583/t0(0) o3->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/400 e 1 to 0 dl 1394011647 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:26:40 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 04:26:40 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 04:26:40 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:27:00 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 04:27:01 dc2oss03 kernel: : LustreError: 16753:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880651aac800 x1461680408013350/t0(0) o3->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/400 e 1 to 0 dl 1394011678 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:27:01 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 04:27:11 dc2oss09 kernel: : Lustre: 16472:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394011631 last 1394011631 Mar 5 04:27:11 dc2oss09 kernel: : Lustre: 16472:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 04:27:12 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 04:27:12 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 04:27:36 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 04:27:36 dc2oss14 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:27:57 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394011677 last 1394011677 Mar 5 04:27:57 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 04:27:53 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:27:53 dc2mds01 kernel: : Lustre: Skipped 18 previous similar messages Mar 5 04:27:53 dc2mds01 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394011673 last 1394011673 Mar 5 04:27:53 dc2mds01 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 18 previous similar messages Mar 5 04:28:08 dc2oss12 kernel: : Lustre: 16158:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394011688 last 1394011688 Mar 5 04:28:08 dc2oss12 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394011688 last 1394011688 Mar 5 04:28:08 dc2oss12 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 04:28:11 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394011691 last 1394011691 Mar 5 04:28:11 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:28:19 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394011699 last 1394011699 Mar 5 04:28:19 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 04:28:59 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:28:59 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 04:28:59 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394011739 last 1394011739 Mar 5 04:28:59 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 04:28:44 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394011724 last 1394011724 Mar 5 04:28:44 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394011724 last 1394011724 Mar 5 04:28:44 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 04:29:03 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:29:03 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:29:03 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394011743 last 1394011743 Mar 5 04:29:03 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 04:28:59 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394011739 last 1394011739 Mar 5 04:28:59 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 04:29:02 dc2oss06 kernel: : Lustre: 14459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394011742 last 1394011742 Mar 5 04:29:02 dc2oss06 kernel: : Lustre: 14459:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 04:29:07 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 04:29:07 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:28:59 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f95a:0x88:0x0]] [valid 68719476736] Mar 5 04:28:59 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:29:12 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881825b13400 cur 1394011752 last 1394011752 Mar 5 04:29:12 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 04:29:14 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394011754 last 1394011754 Mar 5 04:29:14 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 04:29:30 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:29:30 dc2oss04 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 04:29:35 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394011775 last 1394011775 Mar 5 04:29:35 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 04:29:38 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 04:29:38 dc2oss08 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 04:29:44 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:29:44 dc2oss11 kernel: : Lustre: Skipped 131 previous similar messages Mar 5 04:30:10 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:30:10 dc2oss16 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 04:30:13 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:30:13 dc2oss07 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 04:30:13 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1394011813 last 1394011813 Mar 5 04:30:13 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 04:30:40 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 04:30:40 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 04:30:45 dc2oss01 kernel: : Lustre: dc2-OST0005: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:30:45 dc2oss01 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 04:30:45 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0005: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37715128 exp ffff880be6c96400 cur 1394011845 last 1394011845 Mar 5 04:30:45 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 04:31:15 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 04:31:15 dc2oss05 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:31:22 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:31:22 dc2oss14 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 04:31:29 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 04:31:29 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:32:13 dc2oss09 kernel: : Lustre: dc2-OST0057: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:32:13 dc2oss09 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 04:32:48 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:32:48 dc2oss13 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 04:32:54 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:32:54 dc2oss12 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 04:32:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 04:32:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 13 previous similar messages Mar 5 04:32:45 dc2mds01 kernel: : Lustre: 14556:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x39f:0x0]] [valid 68719476736] Mar 5 04:32:45 dc2mds01 kernel: : Lustre: 14556:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:33:07 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:33:07 dc2oss15 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 04:33:26 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:33:26 dc2oss06 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 04:33:16 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232021897852, new: 232021897851 replay: 0. see LU-617. Mar 5 04:34:21 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 04:34:21 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:33:52 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x3ae:0x0]] [valid 68719476736] Mar 5 04:33:52 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:34:22 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:34:22 dc2oss10 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:34:23 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 04:34:23 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:34:24 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:34:24 dc2oss05 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:34:20 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 04:34:20 dc2oss01 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 04:34:27 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 04:34:27 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:34:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 04:34:42 dc2oss09 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 04:35:58 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 04:35:58 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 04:35:55 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 04:35:55 dc2oss07 kernel: : LustreError: Skipped 5 previous similar messages Mar 5 04:36:03 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:36:03 dc2oss08 kernel: : Lustre: Skipped 44 previous similar messages Mar 5 04:36:03 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394012163 last 1394012163 Mar 5 04:36:03 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 44 previous similar messages Mar 5 04:36:02 dc2mds01 kernel: : Lustre: 10017:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x3bf:0x0]] [valid 68719476736] Mar 5 04:36:02 dc2mds01 kernel: : Lustre: 10017:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:36:46 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 04:36:46 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 04:37:07 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 04:37:07 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:37:14 dc2oss09 kernel: : Lustre: 16375:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394012234 last 1394012234 Mar 5 04:37:14 dc2oss09 kernel: : Lustre: 16375:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 04:37:52 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 04:37:52 dc2oss14 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 04:37:53 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 04:37:53 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:38:12 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394012292 last 1394012292 Mar 5 04:38:12 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 04:38:03 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394012283 last 1394012283 Mar 5 04:38:03 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 135 previous similar messages Mar 5 04:37:53 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 04:37:53 dc2mds01 kernel: : Lustre: Skipped 18 previous similar messages Mar 5 04:37:53 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394012273 last 1394012273 Mar 5 04:37:53 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 18 previous similar messages Mar 5 04:38:12 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394012292 last 1394012292 Mar 5 04:38:12 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 04:38:19 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394012299 last 1394012299 Mar 5 04:38:19 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:39:01 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:39:01 dc2oss03 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 04:39:01 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394012341 last 1394012341 Mar 5 04:39:01 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 04:38:46 dc2oss15 kernel: : Lustre: 15757:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394012326 last 1394012326 Mar 5 04:38:46 dc2oss15 kernel: : Lustre: 15757:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 04:39:05 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:39:05 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:39:05 dc2oss02 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 04:39:05 dc2oss02 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 04:39:05 dc2oss02 kernel: : Lustre: 16438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394012345 last 1394012345 Mar 5 04:39:05 dc2oss02 kernel: : Lustre: 16418:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394012345 last 1394012345 Mar 5 04:39:05 dc2oss02 kernel: : Lustre: 16418:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 04:39:09 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394012349 last 1394012349 Mar 5 04:39:09 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 04:39:11 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394012351 last 1394012351 Mar 5 04:39:11 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 04:39:15 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394012355 last 1394012355 Mar 5 04:39:15 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 04:39:20 dc2oss11 kernel: : Lustre: 16223:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881825b13400 cur 1394012360 last 1394012360 Mar 5 04:39:20 dc2oss11 kernel: : Lustre: 16223:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 04:39:24 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0095_UUID' is not available for connect (no target) Mar 5 04:39:24 dc2oss16 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 04:39:31 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:39:31 dc2oss04 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 04:39:36 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394012376 last 1394012376 Mar 5 04:39:36 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 04:39:46 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:39:46 dc2oss11 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 04:40:11 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:40:11 dc2oss16 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 04:40:23 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 04:40:23 dc2oss08 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 04:40:33 dc2oss07 kernel: : Lustre: dc2-OST0048: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:40:33 dc2oss07 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 04:40:33 dc2oss07 kernel: : Lustre: 14421:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0048: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0d048000 cur 1394012433 last 1394012433 Mar 5 04:40:33 dc2oss07 kernel: : Lustre: 14421:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 04:40:23 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x26:0x0]] [valid 68719476736] Mar 5 04:40:23 dc2mds01 kernel: : Lustre: 1367:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:40:41 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 04:40:41 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 04:40:45 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:40:45 dc2oss01 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 04:40:45 dc2oss01 kernel: : Lustre: 15356:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394012445 last 1394012445 Mar 5 04:40:45 dc2oss01 kernel: : Lustre: 15356:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 04:41:32 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:41:32 dc2oss14 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:41:42 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 04:41:42 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 04:41:46 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 04:41:46 dc2oss04 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 04:42:15 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:42:15 dc2oss09 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 04:42:54 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:42:54 dc2oss13 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 04:42:54 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:42:54 dc2oss12 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:43:11 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:43:11 dc2oss15 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 04:43:27 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:43:27 dc2oss06 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 04:43:29 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232023352573, new: 232023352572 replay: 0. see LU-617. Mar 5 04:43:59 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 04:43:59 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 29 previous similar messages Mar 5 04:44:24 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:44:24 dc2oss10 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 04:44:26 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 04:44:26 dc2oss01 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:44:28 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:44:28 dc2oss05 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 04:44:35 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 04:44:35 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 04:44:44 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 04:44:44 dc2oss06 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 04:46:09 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0005_UUID' is not available for connect (no target) Mar 5 04:46:09 dc2oss02 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 04:46:23 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:46:23 dc2oss08 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 04:46:23 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394012783 last 1394012783 Mar 5 04:46:23 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 04:47:06 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 04:47:06 dc2oss03 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 04:47:15 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 04:47:15 dc2oss13 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:47:19 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394012839 last 1394012839 Mar 5 04:47:19 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 04:47:25 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 04:47:25 dc2oss07 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:47:33 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 04:47:33 dc2oss12 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 04:48:01 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 04:48:01 dc2oss14 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 04:48:09 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 04:48:09 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 04:48:09 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394012889 last 1394012889 Mar 5 04:48:09 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 04:48:17 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394012897 last 1394012897 Mar 5 04:48:17 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 04:48:19 dc2oss04 kernel: : Lustre: 16374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394012899 last 1394012899 Mar 5 04:48:19 dc2oss04 kernel: : Lustre: 16374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 74 previous similar messages Mar 5 04:48:43 dc2oss16 kernel: : Lustre: 17368:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394012923 last 1394012923 Mar 5 04:48:43 dc2oss16 kernel: : Lustre: 17368:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 04:48:45 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 04:48:45 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:49:08 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:49:08 dc2oss03 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 04:49:08 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394012948 last 1394012948 Mar 5 04:49:08 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 04:48:53 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0096: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c267d7c00 cur 1394012933 last 1394012933 Mar 5 04:48:53 dc2oss15 kernel: : Lustre: 15842:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 04:48:54 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 04:48:54 dc2mds01 kernel: : Lustre: Skipped 45 previous similar messages Mar 5 04:48:54 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394012934 last 1394012934 Mar 5 04:48:54 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 45 previous similar messages Mar 5 04:49:11 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394012951 last 1394012951 Mar 5 04:49:11 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 04:49:20 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394012960 last 1394012960 Mar 5 04:49:20 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 04:49:31 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:49:31 dc2oss02 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 04:49:31 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394012971 last 1394012971 Mar 5 04:49:31 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 04:49:20 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394012960 last 1394012960 Mar 5 04:49:20 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 04:49:29 dc2oss11 kernel: : Lustre: 15386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394012969 last 1394012969 Mar 5 04:49:29 dc2oss11 kernel: : Lustre: 15386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 04:49:32 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 04:49:32 dc2oss16 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 04:49:31 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:49:31 dc2oss04 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 04:49:25 dc2mds01 kernel: : Lustre: 1373:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f956:0x3d8:0x0]] [valid 68719476736] Mar 5 04:49:25 dc2mds01 kernel: : Lustre: 1373:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 04:49:39 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394012979 last 1394012979 Mar 5 04:49:39 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 04:49:53 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:49:53 dc2oss11 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:50:27 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 04:50:27 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 04:50:38 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:50:38 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 04:50:38 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394013038 last 1394013038 Mar 5 04:50:38 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 04:50:50 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:50:50 dc2oss16 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 04:50:56 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 04:50:56 dc2oss10 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:50:49 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:50:49 dc2oss01 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 04:50:49 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394013049 last 1394013049 Mar 5 04:50:49 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 04:51:33 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:51:33 dc2oss14 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:52:02 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 04:52:02 dc2oss04 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 04:52:07 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 04:52:07 dc2oss05 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 04:52:17 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:52:17 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 04:52:57 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:52:57 dc2oss13 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 04:52:59 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:52:59 dc2oss12 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 04:53:13 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:53:13 dc2oss15 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 04:53:35 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:53:35 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 04:53:51 dc2mds01 kernel: : Lustre: 1375:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x3c:0x0]] [valid 68719476736] Mar 5 04:53:51 dc2mds01 kernel: : Lustre: 1375:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 3 previous similar messages Mar 5 04:54:26 dc2oss10 kernel: : Lustre: dc2-OST0067: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:54:26 dc2oss10 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 04:54:31 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:54:31 dc2oss05 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 04:54:23 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 04:54:23 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 34 previous similar messages Mar 5 04:54:39 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 04:54:39 dc2oss15 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 04:54:36 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 04:54:36 dc2oss01 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 04:55:10 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 04:55:10 dc2oss06 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 04:55:26 dc2oss16 kernel: : Lustre: dc2-OST009f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 04:55:26 dc2oss16 kernel: : LustreError: 15946:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880a6e461000 x1461680324169040/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013372 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:26 dc2oss16 kernel: : Lustre: dc2-OST009f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:25 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 04:55:25 dc2oss07 kernel: : LustreError: 25221:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bda821000 x1461680324169080/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013377 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:25 dc2oss07 kernel: : LustreError: 25221:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 04:55:25 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:25 dc2oss07 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 04:55:26 dc2oss07 kernel: : LustreError: 14532:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88095503e800 x1461680324169087/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013378 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:26 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:26 dc2oss07 kernel: : LustreError: 6362:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880082492000 x1461680324169065/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013375 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:26 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:27 dc2oss08 kernel: : Lustre: dc2-OST004b: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 04:55:28 dc2oss08 kernel: : LustreError: 27734:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8804bcdf3c00 x1461680324168817/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013350 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:28 dc2oss08 kernel: : Lustre: dc2-OST004b: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:28 dc2oss08 kernel: : LustreError: 14652:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88008f514c00 x1461680324169039/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394013350 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:28 dc2oss08 kernel: : Lustre: dc2-OST004b: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:41 dc2oss06 kernel: : Lustre: dc2-OST003c: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 8 active RPCs Mar 5 04:55:41 dc2oss06 kernel: : LustreError: 6968:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88050eefc050 x1461680324169656/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 0 to 0 dl 1394013371 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:41 dc2oss06 kernel: : LustreError: 6968:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 04:55:41 dc2oss06 kernel: : Lustre: dc2-OST003c: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:55:41 dc2oss06 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 04:55:56 dc2oss08 kernel: : Lustre: dc2-OST004d: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 04:55:56 dc2oss08 kernel: : LustreError: 14646:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88077e074400 x1461680324169046/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394013442 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:55:56 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:56:10 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 04:56:10 dc2oss02 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 04:56:23 dc2oss05 kernel: : Lustre: dc2-OST002f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 04:56:23 dc2oss05 kernel: : LustreError: 25054:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8804caea7400 x1461680324169045/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394013486 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:56:23 dc2oss05 kernel: : LustreError: 25054:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 04:56:23 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:56:23 dc2oss05 kernel: : Lustre: Skipped 1 previous similar message Mar 5 04:56:34 dc2oss11 kernel: : Lustre: dc2-OST0070: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 04:56:34 dc2oss11 kernel: : LustreError: 25399:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd4d35400 x1461680324169059/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394013443 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:56:34 dc2oss11 kernel: : LustreError: 25399:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 04:56:34 dc2oss11 kernel: : Lustre: dc2-OST0070: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:56:34 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 04:56:35 dc2oss11 kernel: : LustreError: 8633:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd3dd1000 x1461680324169082/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394013446 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:56:35 dc2oss11 kernel: : LustreError: 8633:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 04:56:27 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:56:27 dc2oss08 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 04:56:27 dc2oss08 kernel: : Lustre: 16172:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394013387 last 1394013387 Mar 5 04:56:27 dc2oss08 kernel: : Lustre: 16172:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 04:57:22 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 04:57:22 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 04:57:19 dc2oss16 kernel: : Lustre: dc2-OST009f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 04:57:19 dc2oss16 kernel: : LustreError: 27756:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88180a8aa050 x1461680324169665/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 0 to 0 dl 1394013442 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 04:57:19 dc2oss16 kernel: : Lustre: dc2-OST009f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 04:57:20 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008b_UUID' is not available for connect (no target) Mar 5 04:57:20 dc2oss13 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 04:57:22 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394013442 last 1394013442 Mar 5 04:57:22 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 04:57:34 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 04:57:34 dc2oss12 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 04:58:14 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 04:58:14 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 04:58:19 dc2oss13 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1394013499 last 1394013499 Mar 5 04:58:19 dc2oss13 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 04:58:20 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394013500 last 1394013500 Mar 5 04:58:20 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 04:58:21 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394013501 last 1394013501 Mar 5 04:58:21 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 04:58:47 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05c45400 cur 1394013527 last 1394013527 Mar 5 04:58:47 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 04:59:11 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:59:11 dc2oss03 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 04:59:11 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394013551 last 1394013551 Mar 5 04:59:11 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 04:58:59 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 04:58:59 dc2oss14 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 04:58:59 dc2oss07 kernel: : Lustre: Service thread pid 25162 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 04:58:59 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 04:58:59 dc2oss07 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 04:58:59 dc2oss07 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 04:58:59 dc2oss07 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 04:58:59 dc2oss07 kernel: : LustreError: dumping log to /tmp/lustre-log.1394013539.25162 Mar 5 04:59:07 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0097: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf83cf000 cur 1394013547 last 1394013547 Mar 5 04:59:07 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 04:59:22 dc2oss10 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394013562 last 1394013562 Mar 5 04:59:22 dc2oss10 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 04:59:22 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394013562 last 1394013562 Mar 5 04:59:22 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 04:59:35 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:59:35 dc2oss02 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 04:59:35 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394013575 last 1394013575 Mar 5 04:59:35 dc2oss02 kernel: : Lustre: 16365:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 04:59:31 dc2oss11 kernel: : Lustre: 15522:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394013571 last 1394013571 Mar 5 04:59:31 dc2oss11 kernel: : Lustre: 15522:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 144 previous similar messages Mar 5 04:59:29 dc2oss06 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394013569 last 1394013569 Mar 5 04:59:29 dc2oss06 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 04:59:37 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 04:59:37 dc2oss09 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 04:59:32 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:59:32 dc2oss04 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 04:59:39 dc2oss05 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0030: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be2951800 cur 1394013579 last 1394013579 Mar 5 04:59:39 dc2oss05 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 04:59:45 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009a_UUID' is not available for connect (no target) Mar 5 04:59:45 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 04:59:40 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:59:40 dc2mds01 kernel: : Lustre: Skipped 41 previous similar messages Mar 5 04:59:40 dc2mds01 kernel: : Lustre: 11166:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394013580 last 1394013580 Mar 5 04:59:40 dc2mds01 kernel: : Lustre: 11166:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 41 previous similar messages Mar 5 04:59:55 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 04:59:55 dc2oss11 kernel: : Lustre: Skipped 148 previous similar messages Mar 5 05:00:29 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 05:00:29 dc2oss08 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 05:00:38 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:00:38 dc2oss07 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 05:00:38 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394013638 last 1394013638 Mar 5 05:00:38 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 05:00:50 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:00:50 dc2oss16 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 05:00:49 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:00:49 dc2oss01 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 05:00:49 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394013649 last 1394013649 Mar 5 05:00:49 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 05:01:07 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 05:01:07 dc2oss10 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 05:01:22 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 05:01:22 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:01:34 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:01:34 dc2oss14 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 05:02:14 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 05:02:14 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 05:02:10 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 05:02:10 dc2oss04 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 05:02:27 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:02:27 dc2oss09 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 05:02:57 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:02:57 dc2oss13 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 05:03:00 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:03:00 dc2oss12 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 05:03:14 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:03:14 dc2oss15 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 05:03:42 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:03:42 dc2oss06 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 05:04:33 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:04:33 dc2oss05 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 05:04:36 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:04:36 dc2oss10 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 05:05:04 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 05:05:04 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 05:05:13 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 05:05:13 dc2oss06 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 05:05:28 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 05:05:28 dc2oss01 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 05:05:40 dc2mds01 kernel: : Lustre: 11130:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f958:0x4e9:0x0]] [valid 68719476736] Mar 5 05:05:40 dc2mds01 kernel: : Lustre: 11130:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 05:06:20 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 05:06:20 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:06:27 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:06:27 dc2oss08 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 05:06:27 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394013987 last 1394013987 Mar 5 05:06:27 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 05:07:38 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 05:07:38 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:07:30 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394014050 last 1394014050 Mar 5 05:07:30 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 05:08:09 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 05:08:09 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 10 previous similar messages Mar 5 05:08:21 dc2oss13 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394014101 last 1394014101 Mar 5 05:08:21 dc2oss13 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 05:08:29 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 05:08:29 dc2oss13 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 05:08:21 dc2oss12 kernel: : Lustre: 16143:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394014101 last 1394014101 Mar 5 05:08:21 dc2oss12 kernel: : Lustre: 16143:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:08:23 dc2oss04 kernel: : Lustre: 16550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0022: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cdc000 cur 1394014103 last 1394014103 Mar 5 05:08:23 dc2oss04 kernel: : Lustre: 16550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 05:09:19 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:09:19 dc2oss03 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 05:09:19 dc2oss03 kernel: : Lustre: 16678:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394014159 last 1394014159 Mar 5 05:09:19 dc2oss03 kernel: : Lustre: 16678:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 05:08:51 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 05:08:51 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 05:08:55 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394014135 last 1394014135 Mar 5 05:08:55 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:09:07 dc2oss15 kernel: : Lustre: 15654:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394014147 last 1394014147 Mar 5 05:09:07 dc2oss15 kernel: : Lustre: 15654:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 05:09:22 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394014162 last 1394014162 Mar 5 05:09:22 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 05:09:14 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 05:09:14 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 05:09:34 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394014174 last 1394014174 Mar 5 05:09:34 dc2oss06 kernel: : Lustre: 15949:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 05:09:26 dc2oss14 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394014166 last 1394014166 Mar 5 05:09:26 dc2oss14 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 05:09:39 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:09:39 dc2oss02 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 05:09:39 dc2oss02 kernel: : Lustre: 15934:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1394014179 last 1394014179 Mar 5 05:09:39 dc2oss02 kernel: : Lustre: 15934:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 05:09:38 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394014178 last 1394014178 Mar 5 05:09:38 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 05:09:34 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:09:34 dc2oss04 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 05:09:39 dc2oss05 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394014179 last 1394014179 Mar 5 05:09:39 dc2oss05 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:09:44 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 05:09:44 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:09:49 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 05:09:49 dc2mds01 kernel: : Lustre: Skipped 11 previous similar messages Mar 5 05:09:49 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394014189 last 1394014189 Mar 5 05:09:49 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages Mar 5 05:10:08 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 05:10:08 dc2oss16 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 05:10:06 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:10:06 dc2oss11 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 05:10:38 dc2oss07 kernel: : Lustre: dc2-OST0044: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:10:38 dc2oss07 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 05:10:38 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12508400 cur 1394014238 last 1394014238 Mar 5 05:10:38 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 05:10:42 dc2oss07 kernel: : Lustre: 6390:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-303), not sending early reply Mar 5 05:10:42 dc2oss07 kernel: : Lustre: 6390:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Mar 5 05:10:47 dc2oss07 kernel: : LustreError: 25162:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 908+0s req@ffff8805d880b400 x1461680324169655/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 4 to 0 dl 1394014247 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 05:10:47 dc2oss07 kernel: : LustreError: 25162:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 2 previous similar messages Mar 5 05:10:47 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 05:10:47 dc2oss07 kernel: : Lustre: Service thread pid 25162 completed after 908.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 05:10:47 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 05:10:56 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:10:56 dc2oss16 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 05:10:49 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:10:49 dc2oss01 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 05:10:49 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394014249 last 1394014249 Mar 5 05:10:49 dc2oss01 kernel: : Lustre: 15504:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 05:11:04 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 05:11:04 dc2oss08 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 05:11:24 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 05:11:24 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:11:25 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 05:11:25 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 05:11:29 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 05:11:29 dc2oss10 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 05:11:34 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:11:34 dc2oss14 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 05:12:17 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 05:12:17 dc2oss05 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 05:12:27 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:12:27 dc2oss09 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 05:12:31 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001a_UUID' is not available for connect (no target) Mar 5 05:12:31 dc2oss04 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 05:12:57 dc2oss13 kernel: : Lustre: dc2-OST007e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:12:57 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 05:13:13 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:13:13 dc2oss12 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 05:13:15 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:13:15 dc2oss15 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 05:13:43 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:13:43 dc2oss06 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 05:13:59 dc2mds01 kernel: : Lustre: 5937:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x61:0x0]] [valid 68719476736] Mar 5 05:13:59 dc2mds01 kernel: : Lustre: 5937:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 05:14:04 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232027586176, new: 232027586175 replay: 0. see LU-617. Mar 5 05:14:34 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:14:34 dc2oss05 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 05:14:39 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:14:39 dc2oss10 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 05:15:48 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 05:15:48 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 05:15:51 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 05:15:51 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 05:16:05 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 05:16:05 dc2oss15 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 05:16:42 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 05:16:42 dc2oss02 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:16:42 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:16:42 dc2oss08 kernel: : Lustre: Skipped 55 previous similar messages Mar 5 05:16:42 dc2oss08 kernel: : Lustre: 15858:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394014602 last 1394014602 Mar 5 05:16:42 dc2oss08 kernel: : Lustre: 15858:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 55 previous similar messages Mar 5 05:17:43 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 05:17:43 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:17:36 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394014656 last 1394014656 Mar 5 05:17:36 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 05:18:10 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 05:18:10 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 18 previous similar messages Mar 5 05:18:24 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394014704 last 1394014704 Mar 5 05:18:24 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 05:18:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 05:18:33 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:18:24 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394014704 last 1394014704 Mar 5 05:18:24 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 05:18:24 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bffed2000 cur 1394014704 last 1394014703 Mar 5 05:18:24 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 05:18:59 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818079be800 cur 1394014739 last 1394014739 Mar 5 05:18:59 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 05:19:24 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:19:24 dc2oss03 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 05:19:24 dc2oss03 kernel: : Lustre: 17492:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394014764 last 1394014764 Mar 5 05:19:24 dc2oss03 kernel: : Lustre: 17492:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 05:19:17 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0080_UUID' is not available for connect (no target) Mar 5 05:19:17 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 05:19:34 dc2oss06 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394014774 last 1394014774 Mar 5 05:19:34 dc2oss06 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 05:19:28 dc2oss10 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394014768 last 1394014768 Mar 5 05:19:28 dc2oss10 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 05:19:28 dc2oss14 kernel: : Lustre: 14586:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394014768 last 1394014768 Mar 5 05:19:28 dc2oss14 kernel: : Lustre: 14586:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 05:19:30 dc2oss15 kernel: : Lustre: 15755:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0098: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881807acd800 cur 1394014770 last 1394014770 Mar 5 05:19:30 dc2oss15 kernel: : Lustre: 15755:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 05:19:44 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:19:44 dc2oss02 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 05:19:44 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394014784 last 1394014784 Mar 5 05:19:44 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:19:37 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:19:37 dc2oss04 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 05:19:42 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394014782 last 1394014782 Mar 5 05:19:42 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 05:19:45 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 05:19:45 dc2oss09 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 05:20:05 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394014805 last 1394014805 Mar 5 05:20:05 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 05:19:51 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:19:51 dc2mds01 kernel: : Lustre: Skipped 30 previous similar messages Mar 5 05:19:51 dc2mds01 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394014791 last 1394014791 Mar 5 05:19:51 dc2mds01 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 30 previous similar messages Mar 5 05:20:25 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:20:25 dc2oss11 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 05:20:32 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 05:20:32 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:20:41 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 05:20:41 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:20:49 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:20:49 dc2oss07 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 05:20:49 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394014849 last 1394014849 Mar 5 05:20:49 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 05:20:56 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:20:56 dc2oss16 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 05:20:55 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:20:55 dc2oss01 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 05:20:55 dc2oss01 kernel: : Lustre: 15386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394014855 last 1394014855 Mar 5 05:20:55 dc2oss01 kernel: : Lustre: 15386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 05:21:07 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 05:21:07 dc2oss08 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:21:37 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007d_UUID' is not available for connect (no target) Mar 5 05:21:37 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:21:39 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:21:39 dc2oss14 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 05:21:47 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 05:21:47 dc2oss10 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 05:21:53 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 05:21:53 dc2oss07 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 05:22:35 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 05:22:35 dc2oss05 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:22:36 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:22:36 dc2oss09 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 05:22:57 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:22:57 dc2oss13 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 05:23:03 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 05:23:03 dc2oss04 kernel: : LustreError: Skipped 48 previous similar messages Mar 5 05:23:20 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:23:20 dc2oss15 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 05:23:17 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:23:17 dc2oss12 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 05:23:43 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:23:43 dc2oss06 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 05:24:41 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:24:41 dc2oss05 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 05:24:53 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:24:53 dc2oss10 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 05:25:38 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x8b:0x0]] [valid 68719476736] Mar 5 05:25:38 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 05:26:05 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 05:26:05 dc2oss15 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:26:06 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 05:26:06 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 05:25:58 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 05:25:58 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 05:26:46 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:26:46 dc2oss08 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 05:26:46 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394015206 last 1394015206 Mar 5 05:26:46 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 05:27:12 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 05:27:12 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:28:02 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 05:28:02 dc2oss03 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 05:27:43 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394015263 last 1394015263 Mar 5 05:27:43 dc2oss09 kernel: : Lustre: 16298:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 05:28:25 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394015305 last 1394015305 Mar 5 05:28:25 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 05:28:31 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394015311 last 1394015311 Mar 5 05:28:31 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 05:28:26 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394015306 last 1394015306 Mar 5 05:28:26 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:28:39 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 05:28:39 dc2oss13 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 05:28:33 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 05:28:33 dc2mds01 kernel: : LustreError: 9759:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 05:29:03 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394015343 last 1394015343 Mar 5 05:29:03 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 05:29:37 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:29:37 dc2oss03 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 05:29:37 dc2oss03 kernel: : Lustre: 16938:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0df95000 cur 1394015377 last 1394015377 Mar 5 05:29:37 dc2oss03 kernel: : Lustre: 16938:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 05:29:34 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394015374 last 1394015374 Mar 5 05:29:34 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 05:29:45 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:29:45 dc2oss02 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 05:29:45 dc2oss02 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394015385 last 1394015385 Mar 5 05:29:45 dc2oss02 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 05:29:35 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 05:29:35 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:29:39 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37972127 exp ffff880c08fefc00 cur 1394015379 last 1394015379 Mar 5 05:29:39 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 05:29:35 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394015375 last 1394015375 Mar 5 05:29:35 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394015375 last 1394015375 Mar 5 05:29:35 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:29:35 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:29:43 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394015383 last 1394015383 Mar 5 05:29:43 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:29:54 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394015394 last 1394015394 Mar 5 05:29:54 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:29:54 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:29:54 dc2oss04 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 05:30:05 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0063_UUID' is not available for connect (no target) Mar 5 05:30:05 dc2oss09 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:30:08 dc2oss11 kernel: : Lustre: 16230:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394015408 last 1394015408 Mar 5 05:30:08 dc2oss11 kernel: : Lustre: 16230:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:29:54 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 05:29:54 dc2mds01 kernel: : Lustre: Skipped 38 previous similar messages Mar 5 05:29:54 dc2mds01 kernel: : Lustre: 9759:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394015394 last 1394015394 Mar 5 05:29:54 dc2mds01 kernel: : Lustre: 9759:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 38 previous similar messages Mar 5 05:30:45 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:30:45 dc2oss11 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 05:30:49 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:30:49 dc2oss07 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 05:30:49 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394015449 last 1394015449 Mar 5 05:30:49 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:30:57 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:30:57 dc2oss16 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 05:31:05 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 05:31:05 dc2oss16 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 05:31:08 dc2xfer04 kernel: : Lustre: 2713:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:08 dc2xfer04 kernel: : Lustre: 2713:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:08 dc2xfer04 kernel: : Lustre: 2713:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 05:31:08 dc2xfer04 kernel: : Lustre: 2713:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 05:31:04 dc2oss13 kernel: : Lustre: dc2-OST0086: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 05:31:04 dc2oss13 kernel: : LustreError: 6383:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8801f5b99800 x1461679862671777/t0(0) o4->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/416 e 1 to 0 dl 1394015492 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 05:31:04 dc2oss13 kernel: : LustreError: 6383:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 05:31:04 dc2oss13 kernel: : Lustre: dc2-OST0086: Bulk IO write error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 05:31:04 dc2oss13 kernel: : Lustre: Skipped 1 previous similar message Mar 5 05:31:04 dc2oss13 kernel: : LustreError: 24945:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc5918c00 x1461679862671765/t0(0) o4->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/416 e 1 to 0 dl 1394015491 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 05:31:04 dc2oss13 kernel: : LustreError: 24945:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 05:31:04 dc2oss13 kernel: : Lustre: dc2-OST0086: Bulk IO write error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 05:31:04 dc2oss13 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 05:31:08 dc2xfer03 kernel: : Lustre: 2643:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:08 dc2xfer03 kernel: : Lustre: 2643:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:09 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:09 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:09 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 8 previous similar messages Mar 5 05:31:09 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 8 previous similar messages Mar 5 05:31:09 dc2xfer02 kernel: : Lustre: 2706:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:09 dc2xfer02 kernel: : Lustre: 2706:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.13@o2ib Mar 5 05:31:09 dc2xfer02 kernel: : Lustre: 2706:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 05:31:09 dc2xfer02 kernel: : Lustre: 2706:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 10 previous similar messages Mar 5 05:31:11 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 05:31:11 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 05:31:04 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:31:04 dc2oss01 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 05:31:04 dc2oss01 kernel: : Lustre: 15661:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394015464 last 1394015464 Mar 5 05:31:04 dc2oss01 kernel: : Lustre: 15661:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:31:21 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0069_UUID' is not available for connect (no target) Mar 5 05:31:21 dc2oss12 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 05:31:45 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:31:45 dc2oss14 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 05:31:53 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 05:31:53 dc2oss10 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 05:31:56 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 05:31:56 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:32:01 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 05:32:01 dc2oss07 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:32:37 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:32:37 dc2oss09 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 05:33:00 dc2oss13 kernel: : Lustre: dc2-OST007f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:33:00 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:33:00 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 05:33:00 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 05:33:23 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:33:23 dc2oss15 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 05:33:21 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001d_UUID' is not available for connect (no target) Mar 5 05:33:21 dc2oss04 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:33:28 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:33:28 dc2oss12 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 05:33:31 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 05:33:31 dc2oss05 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:33:45 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:33:45 dc2oss06 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 05:34:47 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:34:47 dc2oss05 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 05:34:54 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:34:54 dc2oss10 kernel: : Lustre: Skipped 74 previous similar messages Mar 5 05:36:01 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 05:36:01 dc2oss01 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 05:35:59 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0xb0:0x0]] [valid 68719476736] Mar 5 05:35:59 dc2mds01 kernel: : Lustre: 14552:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 05:36:23 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 05:36:23 dc2oss15 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:36:27 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 05:36:27 dc2oss06 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 05:36:48 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:36:48 dc2oss08 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 05:36:48 dc2oss08 kernel: : Lustre: 15866:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394015808 last 1394015808 Mar 5 05:36:48 dc2oss08 kernel: : Lustre: 15866:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 05:37:25 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 05:37:25 dc2oss02 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:38:04 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 05:38:04 dc2oss03 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:37:52 dc2oss09 kernel: : Lustre: 16391:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c00a7e400 cur 1394015872 last 1394015872 Mar 5 05:37:52 dc2oss09 kernel: : Lustre: 16391:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 05:38:31 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394015911 last 1394015911 Mar 5 05:38:31 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 05:38:27 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394015907 last 1394015907 Mar 5 05:38:27 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 05:38:35 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0022: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cdc000 cur 1394015915 last 1394015915 Mar 5 05:38:35 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 05:38:48 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 05:38:48 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:39:04 dc2oss16 kernel: : Lustre: 15731:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394015944 last 1394015944 Mar 5 05:39:04 dc2oss16 kernel: : Lustre: 15731:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 05:39:37 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:39:37 dc2oss03 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 05:39:37 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394015977 last 1394015977 Mar 5 05:39:37 dc2oss03 kernel: : Lustre: 17107:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 05:39:35 dc2oss06 kernel: : Lustre: 14420:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394015975 last 1394015975 Mar 5 05:39:35 dc2oss06 kernel: : Lustre: 14420:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 05:39:46 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:39:46 dc2oss02 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 05:39:46 dc2oss02 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394015986 last 1394015986 Mar 5 05:39:46 dc2oss02 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 05:39:36 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 05:39:36 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:39:40 dc2oss14 kernel: : Lustre: 15564:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394015980 last 1394015980 Mar 5 05:39:40 dc2oss14 kernel: : Lustre: 15564:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 05:39:39 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394015979 last 1394015979 Mar 5 05:39:39 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 05:39:43 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394015983 last 1394015983 Mar 5 05:39:43 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 05:39:54 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:39:54 dc2oss04 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 05:40:00 dc2oss10 kernel: : Lustre: 15827:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394016000 last 1394016000 Mar 5 05:40:00 dc2oss10 kernel: : Lustre: 15827:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 05:40:10 dc2oss11 kernel: : Lustre: 16230:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394016010 last 1394016010 Mar 5 05:40:10 dc2oss11 kernel: : Lustre: 16230:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 05:40:32 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 05:40:32 dc2oss09 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 05:40:38 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 05:40:38 dc2mds01 kernel: : Lustre: Skipped 18 previous similar messages Mar 5 05:40:38 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394016038 last 1394016038 Mar 5 05:40:38 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 18 previous similar messages Mar 5 05:40:38 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 05:40:38 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 17 previous similar messages Mar 5 05:40:51 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:40:51 dc2oss11 kernel: : Lustre: Skipped 136 previous similar messages Mar 5 05:40:58 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:40:58 dc2oss16 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 05:40:58 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:40:58 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 05:40:58 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394016058 last 1394016058 Mar 5 05:40:58 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 05:41:06 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:41:06 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:41:06 dc2oss01 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 05:41:06 dc2oss01 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 05:41:06 dc2oss01 kernel: : Lustre: 15568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394016066 last 1394016066 Mar 5 05:41:06 dc2oss01 kernel: : Lustre: 15568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 05:41:28 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 05:41:28 dc2oss08 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 05:41:43 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 05:41:43 dc2oss12 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 05:41:47 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 05:41:47 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:41:52 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:41:52 dc2oss14 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 05:42:12 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 05:42:12 dc2oss10 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 05:42:12 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 05:42:12 dc2oss07 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 05:42:42 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 05:42:42 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:42:46 dc2oss09 kernel: : Lustre: dc2-OST0057: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:42:46 dc2oss09 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 05:43:04 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:43:04 dc2oss13 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 05:43:24 dc2oss15 kernel: : Lustre: dc2-OST0095: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:43:24 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 05:43:25 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 05:43:25 dc2oss04 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 05:43:34 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:43:34 dc2oss12 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 05:44:10 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:44:10 dc2oss06 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 05:44:21 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 05:44:21 dc2oss05 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 05:44:51 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:44:51 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 05:45:03 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:45:03 dc2oss10 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 05:46:27 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 05:46:27 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 05:46:30 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 05:46:30 dc2oss01 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 05:46:56 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:46:56 dc2oss08 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 05:46:56 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394016416 last 1394016416 Mar 5 05:46:56 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 05:47:32 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 05:47:32 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:47:34 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 05:47:34 dc2oss06 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 05:48:05 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 05:48:05 dc2oss03 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:47:57 dc2oss09 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394016477 last 1394016477 Mar 5 05:47:57 dc2oss09 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 05:48:29 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394016509 last 1394016509 Mar 5 05:48:29 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 05:48:39 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394016519 last 1394016519 Mar 5 05:48:39 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:48:44 dc2oss04 kernel: : Lustre: 16478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1394016524 last 1394016524 Mar 5 05:48:44 dc2oss04 kernel: : Lustre: 16478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 05:48:52 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 05:48:52 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:48:47 dc2mds01 kernel: : Lustre: 14542:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201d6093c:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 05:48:58 dc2mds01 kernel: : Lustre: 14542:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201e2c822:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 05:49:12 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394016552 last 1394016552 Mar 5 05:49:12 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 05:49:54 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:49:54 dc2oss03 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 05:49:54 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394016594 last 1394016594 Mar 5 05:49:54 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 05:49:52 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:49:52 dc2oss02 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 05:49:52 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394016592 last 1394016592 Mar 5 05:49:52 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 05:49:40 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394016580 last 1394016580 Mar 5 05:49:40 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 05:49:40 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1394016580 last 1394016580 Mar 5 05:49:40 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 05:49:41 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0081_UUID' is not available for connect (no target) Mar 5 05:49:41 dc2oss14 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:49:44 dc2oss05 kernel: : Lustre: 15954:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394016584 last 1394016584 Mar 5 05:49:44 dc2oss05 kernel: : Lustre: 15954:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 05:49:48 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1394016588 last 1394016588 Mar 5 05:49:48 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 05:50:04 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394016604 last 1394016604 Mar 5 05:50:04 dc2oss10 kernel: : Lustre: 16525:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 05:50:03 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:50:03 dc2oss04 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 05:49:55 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0xca:0x0]] [valid 68719476736] Mar 5 05:49:55 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 05:50:14 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394016614 last 1394016614 Mar 5 05:50:14 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 05:50:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 05:50:42 dc2oss09 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 05:50:39 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 05:50:39 dc2mds01 kernel: : Lustre: Skipped 38 previous similar messages Mar 5 05:50:39 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394016639 last 1394016639 Mar 5 05:50:39 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 38 previous similar messages Mar 5 05:50:39 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 05:50:39 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 27 previous similar messages Mar 5 05:50:54 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:50:54 dc2oss11 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 05:51:00 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:51:00 dc2oss16 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 05:50:59 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:50:59 dc2oss07 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 05:50:59 dc2oss07 kernel: : Lustre: 16000:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394016659 last 1394016659 Mar 5 05:50:59 dc2oss07 kernel: : Lustre: 16000:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 05:51:09 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:51:09 dc2oss01 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 05:51:09 dc2oss01 kernel: : Lustre: 15568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394016669 last 1394016669 Mar 5 05:51:09 dc2oss01 kernel: : Lustre: 15568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:51:51 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 05:51:51 dc2oss08 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:51:54 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 05:51:54 dc2oss16 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 05:52:08 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:52:08 dc2oss14 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 05:52:18 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 05:52:18 dc2oss10 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 05:52:16 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 05:52:16 dc2oss12 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 05:52:16 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 05:52:16 dc2oss07 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 05:52:43 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 05:52:43 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 05:52:53 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:52:53 dc2oss09 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 05:53:18 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:53:18 dc2oss13 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 05:53:25 dc2oss15 kernel: : Lustre: dc2-OST0095: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:53:25 dc2oss15 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 05:53:46 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 05:53:46 dc2oss04 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 05:53:53 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:53:53 dc2oss12 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 05:54:12 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:54:12 dc2oss06 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 05:54:59 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 05:54:59 dc2oss05 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 05:55:01 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:55:01 dc2oss05 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 05:55:07 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:55:07 dc2oss10 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 05:57:03 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 05:57:03 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:57:29 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a0_UUID' is not available for connect (no target) Mar 5 05:57:29 dc2oss15 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 05:57:29 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:57:29 dc2oss08 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 05:57:29 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394017049 last 1394017049 Mar 5 05:57:29 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 05:57:41 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 05:57:41 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 05:58:05 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394017085 last 1394017085 Mar 5 05:58:05 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 05:58:22 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 05:58:22 dc2oss03 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 05:58:14 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 05:58:14 dc2oss02 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 05:58:33 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394017113 last 1394017113 Mar 5 05:58:33 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:58:24 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232033018402, new: 232033018401 replay: 0. see LU-617. Mar 5 05:58:45 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394017125 last 1394017125 Mar 5 05:58:45 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 05:58:45 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394017125 last 1394017125 Mar 5 05:58:45 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 05:59:03 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 05:59:03 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 05:59:14 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394017154 last 1394017154 Mar 5 05:59:14 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 05:59:54 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:59:54 dc2oss02 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 05:59:54 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394017194 last 1394017194 Mar 5 05:59:54 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 05:59:58 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 05:59:58 dc2oss03 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 05:59:58 dc2oss03 kernel: : Lustre: 17443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394017198 last 1394017198 Mar 5 05:59:58 dc2oss03 kernel: : Lustre: 17443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:59:40 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394017180 last 1394017180 Mar 5 05:59:40 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 05:59:53 dc2oss05 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394017193 last 1394017193 Mar 5 05:59:53 dc2oss05 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 05:59:53 dc2oss14 kernel: : Lustre: 15688:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1394017193 last 1394017193 Mar 5 05:59:53 dc2oss14 kernel: : Lustre: 15688:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 06:00:08 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394017208 last 1394017208 Mar 5 06:00:08 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 06:00:04 dc2oss06 kernel: : Lustre: 14429:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394017204 last 1394017204 Mar 5 06:00:04 dc2oss06 kernel: : Lustre: 14429:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 06:00:05 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:00:05 dc2oss04 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 06:00:20 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394017220 last 1394017220 Mar 5 06:00:20 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 06:00:35 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 06:00:35 dc2oss14 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 06:00:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 06:00:42 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:00:45 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 06:00:45 dc2mds01 kernel: : Lustre: Skipped 31 previous similar messages Mar 5 06:00:45 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394017245 last 1394017245 Mar 5 06:00:45 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages Mar 5 06:00:45 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:00:45 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 21 previous similar messages Mar 5 06:00:58 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:00:58 dc2oss11 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 06:01:06 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:01:06 dc2oss16 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 06:01:10 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:01:10 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 06:01:10 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394017270 last 1394017270 Mar 5 06:01:10 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 06:01:10 dc2oss01 kernel: : Lustre: dc2-OST0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:01:10 dc2oss01 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 06:01:10 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c003ae000 cur 1394017270 last 1394017270 Mar 5 06:01:10 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 06:01:04 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x194:0x0]] [valid 68719476736] Mar 5 06:01:04 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 06:01:59 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009c_UUID' is not available for connect (no target) Mar 5 06:01:59 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:02:10 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:02:10 dc2oss14 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 06:02:29 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0044_UUID' is not available for connect (no target) Mar 5 06:02:29 dc2oss08 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:02:54 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:02:54 dc2oss09 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 06:03:02 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0056_UUID' is not available for connect (no target) Mar 5 06:03:02 dc2oss10 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:03:08 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 06:03:08 dc2oss07 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:03:11 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 06:03:11 dc2oss12 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:03:05 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232033661698, new: 232033661697 replay: 0. see LU-617. Mar 5 06:03:19 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:03:19 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 06:03:23 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 06:03:23 dc2oss11 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:03:29 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:03:29 dc2oss15 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 06:03:56 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 06:03:56 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:04:09 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:04:09 dc2oss12 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 06:04:19 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:04:19 dc2oss06 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 06:05:12 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:05:12 dc2oss10 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 06:05:14 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:05:14 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:05:14 dc2oss05 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 06:05:14 dc2oss05 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 06:05:16 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232033992283, new: 232033992282 replay: 0. see LU-617. Mar 5 06:05:43 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 06:05:43 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 06:07:40 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 06:07:40 dc2oss15 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:07:51 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:07:51 dc2oss08 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 06:07:51 dc2oss08 kernel: : Lustre: 16123:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394017671 last 1394017671 Mar 5 06:07:51 dc2oss08 kernel: : Lustre: 16123:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 06:08:07 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394017687 last 1394017687 Mar 5 06:08:07 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 06:08:02 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002a_UUID' is not available for connect (no target) Mar 5 06:08:02 dc2oss06 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:08:22 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 06:08:22 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:08:46 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 06:08:46 dc2oss03 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 06:08:41 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394017721 last 1394017721 Mar 5 06:08:41 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 06:08:46 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394017726 last 1394017726 Mar 5 06:08:46 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 06:08:48 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394017728 last 1394017728 Mar 5 06:08:48 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 76 previous similar messages Mar 5 06:09:04 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 06:09:04 dc2oss13 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 06:09:00 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 06:09:00 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:09:19 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394017759 last 1394017759 Mar 5 06:09:19 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 06:09:55 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:09:55 dc2oss02 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 06:09:55 dc2oss02 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394017795 last 1394017794 Mar 5 06:09:55 dc2oss02 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 06:10:01 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:10:01 dc2oss03 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 06:10:01 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394017801 last 1394017801 Mar 5 06:10:01 dc2oss03 kernel: : Lustre: 17125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 06:09:52 dc2oss15 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394017792 last 1394017792 Mar 5 06:09:52 dc2oss15 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 06:10:00 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394017800 last 1394017800 Mar 5 06:10:00 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 06:10:06 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394017806 last 1394017806 Mar 5 06:10:06 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 06:10:04 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394017804 last 1394017804 Mar 5 06:10:04 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 06:10:08 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc10e800 cur 1394017808 last 1394017808 Mar 5 06:10:08 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 06:10:07 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:10:07 dc2oss04 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 06:10:22 dc2oss05 kernel: : Lustre: dc2-OST002d: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 06:10:22 dc2oss05 kernel: : LustreError: 25053:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806000b6400 x1461679898543639/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 0 to 0 dl 1394017851 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:22 dc2oss05 kernel: : Lustre: dc2-OST002d: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:23 dc2oss05 kernel: : LustreError: 29830:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8802ce0b0000 x1461679898543649/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 0 to 0 dl 1394017852 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:23 dc2oss05 kernel: : Lustre: dc2-OST002d: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:34 dc2oss02 kernel: : Lustre: dc2-OST000e: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 06:10:35 dc2oss02 kernel: : LustreError: 21102:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf8827050 x1461679898543706/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394017874 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:35 dc2oss02 kernel: : LustreError: 21102:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:10:35 dc2oss02 kernel: : Lustre: dc2-OST000e: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:35 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:10:35 dc2oss02 kernel: : LustreError: 15983:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88079179fc00 x1461679898543690/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394017871 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:35 dc2oss02 kernel: : Lustre: dc2-OST000e: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:34 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff880bf1002c00 cur 1394017834 last 1394017834 Mar 5 06:10:34 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 06:10:31 dc2oss14 kernel: : Lustre: dc2-OST008c: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 06:10:31 dc2oss14 kernel: : LustreError: 22869:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc8714400 x1461679898543685/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394017885 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:31 dc2oss14 kernel: : LustreError: 22869:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:10:31 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:31 dc2oss14 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:10:32 dc2oss14 kernel: : LustreError: 22682:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc359d000 x1461679898543651/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394017884 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:10:32 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 06:10:44 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0080_UUID' is not available for connect (no target) Mar 5 06:10:44 dc2oss14 kernel: : LustreError: Skipped 45 previous similar messages Mar 5 06:10:58 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:10:58 dc2oss11 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 06:11:03 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 06:11:03 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:11:07 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:11:07 dc2oss16 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 06:11:11 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:11:11 dc2oss07 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 06:11:11 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394017871 last 1394017871 Mar 5 06:11:11 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 06:11:03 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:11:03 dc2mds01 kernel: : Lustre: Skipped 22 previous similar messages Mar 5 06:11:03 dc2mds01 kernel: : Lustre: 11140:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394017863 last 1394017863 Mar 5 06:11:03 dc2mds01 kernel: : Lustre: 11140:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 22 previous similar messages Mar 5 06:11:31 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:11:31 dc2oss01 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 06:11:31 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394017891 last 1394017891 Mar 5 06:11:31 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 06:11:52 dc2mds01 kernel: : Lustre: 11178:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f961:0x375d:0x0]] [valid 68719476736] Mar 5 06:11:52 dc2mds01 kernel: : Lustre: 11178:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 13 previous similar messages Mar 5 06:12:32 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009a_UUID' is not available for connect (no target) Mar 5 06:12:32 dc2oss16 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 06:12:26 dc2oss14 kernel: : Lustre: dc2-OST008b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:12:26 dc2oss14 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 06:12:56 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:12:56 dc2oss09 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 06:13:19 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:13:19 dc2oss13 kernel: : Lustre: Skipped 139 previous similar messages Mar 5 06:13:09 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:13:09 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 06:13:23 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 06:13:23 dc2oss08 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 06:13:27 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 06:13:27 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:13:32 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 06:13:32 dc2oss10 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 06:13:33 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:13:33 dc2oss15 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 06:13:52 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 06:13:52 dc2oss11 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 06:14:05 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 06:14:05 dc2oss12 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:14:23 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:14:23 dc2oss12 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 06:14:25 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:14:25 dc2oss06 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 06:14:49 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 06:14:49 dc2oss04 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:15:14 dc2oss05 kernel: : Lustre: dc2-OST002b: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) reconnecting Mar 5 06:15:14 dc2oss05 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 06:15:15 dc2oss10 kernel: : Lustre: dc2-OST0061: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:15:15 dc2oss10 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 06:15:34 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 06:15:34 dc2oss08 kernel: : LustreError: 27731:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd3880000 x1461679859768633/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 0 to 0 dl 1394018160 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:15:34 dc2oss08 kernel: : LustreError: 27731:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:15:34 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 06:15:34 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:15:34 dc2oss08 kernel: : LustreError: 27739:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b75dabc00 x1461679859768853/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 0 to 0 dl 1394018160 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:15:34 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 06:15:39 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 06:15:39 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:16:24 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 06:16:24 dc2oss05 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:17:51 dc2oss08 kernel: : Lustre: dc2-OST0052: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:17:51 dc2oss08 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 06:17:51 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1394018271 last 1394018271 Mar 5 06:17:51 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 06:18:07 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394018287 last 1394018287 Mar 5 06:18:07 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 06:18:09 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 06:18:09 dc2oss06 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 06:18:19 dc2oss02 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 161s: evicting client at 149.165.226.209@tcp ns: filter-dc2-OST000e_UUID lock: ffff8817e944bb40/0x1764d0f189b966a3 lrc: 3/0,0 mode: PR/PR res: 22866010/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xe11b9f3f2c3c69b expref: 7 pid: 15891 timeout 4360811207 Mar 5 06:18:19 dc2oss02 kernel: : LustreError: 15814:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88078eb1ec00 x1461665913939149/t0(0) o105->dc2-OST000e@149.165.226.209@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 Mar 5 06:18:19 dc2oss02 kernel: : LustreError: 15814:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.209@tcp) returned 0 from completion AST ns: filter-dc2-OST000e_UUID lock: ffff880e9c79d240/0x1764d0f189b96951 lrc: 3/0,0 mode: PW/PW res: 22866010/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->16383) flags: 0x0 remote: 0xe11b9f3f2c3d1df expref: 5 pid: 16278 timeout 0 Mar 5 06:18:17 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 06:18:17 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:18:33 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 06:18:33 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:18:41 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394018321 last 1394018321 Mar 5 06:18:41 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 06:19:11 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 06:19:11 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:18:51 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394018331 last 1394018331 Mar 5 06:18:51 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 06:18:50 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394018330 last 1394018330 Mar 5 06:18:50 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 06:19:04 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 06:19:04 dc2oss13 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:19:08 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 06:19:08 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:19:26 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394018366 last 1394018366 Mar 5 06:19:26 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 06:19:19 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232035689201, new: 232035689200 replay: 0. see LU-617. Mar 5 06:19:55 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:19:55 dc2oss02 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 06:19:55 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394018395 last 1394018395 Mar 5 06:19:55 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 06:20:06 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:20:06 dc2oss03 kernel: : Lustre: Skipped 163 previous similar messages Mar 5 06:20:06 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394018406 last 1394018406 Mar 5 06:20:06 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 161 previous similar messages Mar 5 06:19:55 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394018395 last 1394018395 Mar 5 06:19:55 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 137 previous similar messages Mar 5 06:20:01 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394018401 last 1394018401 Mar 5 06:20:01 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 06:20:06 dc2oss06 kernel: : Lustre: 15776:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1394018406 last 1394018406 Mar 5 06:20:06 dc2oss06 kernel: : Lustre: 15776:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 06:20:14 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394018414 last 1394018414 Mar 5 06:20:14 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 06:20:27 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394018427 last 1394018427 Mar 5 06:20:27 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 140 previous similar messages Mar 5 06:20:24 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:20:24 dc2oss04 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 06:20:39 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394018439 last 1394018439 Mar 5 06:20:39 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 06:21:06 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:21:06 dc2oss11 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 06:21:10 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:21:10 dc2oss16 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 06:21:05 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232035953380, new: 232035953379 replay: 0. see LU-617. Mar 5 06:21:24 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:21:24 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 06:21:24 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394018484 last 1394018484 Mar 5 06:21:24 dc2oss07 kernel: : Lustre: 15770:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 06:21:28 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 06:21:28 dc2mds01 kernel: : Lustre: Skipped 31 previous similar messages Mar 5 06:21:28 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394018488 last 1394018488 Mar 5 06:21:28 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages Mar 5 06:21:35 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:21:35 dc2oss01 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 06:21:35 dc2oss01 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394018495 last 1394018495 Mar 5 06:21:35 dc2oss01 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 06:21:47 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 06:21:47 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:21:49 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 06:21:49 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:22:39 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:22:39 dc2oss14 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 06:22:56 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 06:22:56 dc2oss16 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:22:57 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:22:57 dc2oss09 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 06:23:22 dc2oss13 kernel: : Lustre: dc2-OST0085: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:23:22 dc2oss13 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 06:23:33 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 06:23:33 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 06:23:45 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:23:45 dc2oss15 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 06:24:04 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 06:24:04 dc2oss10 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 06:24:04 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 06:24:04 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:23:53 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:23:53 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 29 previous similar messages Mar 5 06:24:25 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:24:25 dc2oss12 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 06:24:32 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:24:32 dc2oss06 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 06:25:22 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:25:22 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 06:25:12 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 06:25:12 dc2oss04 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:25:33 dc2oss10 kernel: : Lustre: dc2-OST0061: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:25:33 dc2oss10 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 06:26:31 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 06:26:31 dc2oss05 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:26:50 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 06:26:50 dc2oss12 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 06:27:19 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004d_UUID' is not available for connect (no target) Mar 5 06:27:19 dc2oss07 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:27:59 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:27:59 dc2oss08 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 06:27:59 dc2oss08 kernel: : Lustre: 16020:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394018879 last 1394018879 Mar 5 06:27:59 dc2oss08 kernel: : Lustre: 16020:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 75 previous similar messages Mar 5 06:28:08 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394018888 last 1394018888 Mar 5 06:28:08 dc2oss09 kernel: : Lustre: 16189:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 06:28:12 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x12a:0x0]] [valid 68719476736] Mar 5 06:28:12 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 06:28:39 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 06:28:39 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:28:53 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394018933 last 1394018933 Mar 5 06:28:53 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1394018933 last 1394018933 Mar 5 06:28:53 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 06:28:53 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 06:28:50 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1394018930 last 1394018930 Mar 5 06:28:50 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 06:28:53 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1394018933 last 1394018933 Mar 5 06:28:53 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 06:29:01 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 06:29:01 dc2oss15 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 06:29:21 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 06:29:21 dc2oss03 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 06:29:25 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 06:29:25 dc2oss02 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 06:29:16 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 06:29:16 dc2oss13 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 06:29:47 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394018987 last 1394018987 Mar 5 06:29:47 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 06:30:11 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:30:11 dc2oss02 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 06:30:11 dc2oss02 kernel: : Lustre: 16491:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394019011 last 1394019011 Mar 5 06:30:11 dc2oss02 kernel: : Lustre: 16491:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 06:30:15 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:30:15 dc2oss03 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 06:30:15 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394019015 last 1394019015 Mar 5 06:30:15 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 06:30:00 dc2oss15 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394019000 last 1394019000 Mar 5 06:30:00 dc2oss15 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 06:30:06 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37972127 exp ffff880c08fefc00 cur 1394019006 last 1394019006 Mar 5 06:30:06 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 86 previous similar messages Mar 5 06:30:09 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394019009 last 1394019009 Mar 5 06:30:09 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 06:30:14 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394019014 last 1394019014 Mar 5 06:30:14 dc2oss10 kernel: : Lustre: 16290:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 06:30:12 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 06:30:12 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:30:24 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:30:24 dc2oss04 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 06:30:32 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394019032 last 1394019032 Mar 5 06:30:32 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 06:30:40 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394019040 last 1394019040 Mar 5 06:30:40 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 06:31:15 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:31:15 dc2oss16 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 06:31:22 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232037226600, new: 232037226599 replay: 0. see LU-617. Mar 5 06:31:30 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 06:31:30 dc2mds01 kernel: : Lustre: Skipped 29 previous similar messages Mar 5 06:31:30 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394019090 last 1394019090 Mar 5 06:31:30 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 29 previous similar messages Mar 5 06:31:41 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:31:41 dc2oss07 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 06:31:41 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394019101 last 1394019101 Mar 5 06:31:41 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 06:31:41 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:31:41 dc2oss11 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 06:31:36 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:31:36 dc2oss01 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 06:31:36 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394019096 last 1394019096 Mar 5 06:31:36 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 06:31:48 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 06:31:48 dc2oss09 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:31:50 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 06:31:50 dc2oss14 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:32:44 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:32:44 dc2oss14 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 06:32:59 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:32:59 dc2oss09 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 06:33:14 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 06:33:14 dc2oss16 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:33:36 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:33:36 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:33:36 dc2oss13 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 06:33:36 dc2oss13 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 06:33:47 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:33:47 dc2oss15 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 06:33:48 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 06:33:48 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 06:34:07 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 06:34:07 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 06:33:56 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:33:56 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 06:34:16 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 06:34:16 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:34:32 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:34:32 dc2oss06 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 06:34:29 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:34:29 dc2oss12 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 06:35:22 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:35:22 dc2oss05 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 06:35:38 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:35:38 dc2oss10 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 06:35:36 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0019_UUID' is not available for connect (no target) Mar 5 06:35:36 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:36:50 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 06:36:50 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 06:37:02 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 06:37:02 dc2oss12 kernel: : LustreError: Skipped 42 previous similar messages Mar 5 06:38:01 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:38:01 dc2oss08 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 06:38:01 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394019481 last 1394019481 Mar 5 06:38:01 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 06:38:09 dc2oss09 kernel: : Lustre: 14563:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394019489 last 1394019489 Mar 5 06:38:09 dc2oss09 kernel: : Lustre: 14563:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 06:38:11 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0050_UUID' is not available for connect (no target) Mar 5 06:38:11 dc2oss07 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 06:38:55 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394019535 last 1394019535 Mar 5 06:38:55 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 06:38:54 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 06:38:54 dc2oss06 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 06:38:44 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x220:0x0]] [valid 68719476736] Mar 5 06:38:44 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 11 previous similar messages Mar 5 06:38:56 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394019536 last 1394019536 Mar 5 06:38:56 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 06:38:59 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394019539 last 1394019539 Mar 5 06:38:59 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 06:39:17 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 06:39:17 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:39:28 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 06:39:28 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:39:23 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 06:39:23 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 06:39:47 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 06:39:47 dc2oss03 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 06:39:57 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394019597 last 1394019597 Mar 5 06:39:57 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 06:40:06 dc2oss15 kernel: : Lustre: 15918:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394019606 last 1394019606 Mar 5 06:40:06 dc2oss15 kernel: : Lustre: 15918:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 06:40:09 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394019609 last 1394019609 Mar 5 06:40:09 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 06:40:09 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1394019609 last 1394019609 Mar 5 06:40:09 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 06:40:24 dc2oss03 kernel: : Lustre: dc2-OST0016: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:40:24 dc2oss03 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 06:40:24 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31136294 exp ffff880c0266f000 cur 1394019624 last 1394019624 Mar 5 06:40:24 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 06:40:16 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:40:16 dc2oss02 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 06:40:16 dc2oss02 kernel: : Lustre: 16323:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394019616 last 1394019616 Mar 5 06:40:16 dc2oss02 kernel: : Lustre: 16323:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 06:40:16 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394019616 last 1394019616 Mar 5 06:40:16 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 06:40:23 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 06:40:23 dc2oss01 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:40:26 dc2oss04 kernel: : Lustre: dc2-OST0022: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:40:26 dc2oss04 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 06:40:33 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394019633 last 1394019633 Mar 5 06:40:33 dc2oss05 kernel: : Lustre: 14630:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 06:40:51 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394019651 last 1394019651 Mar 5 06:40:51 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 06:41:17 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:41:17 dc2oss16 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 06:41:44 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:41:44 dc2oss07 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 06:41:44 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394019704 last 1394019704 Mar 5 06:41:44 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 06:41:44 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:41:44 dc2oss11 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 06:41:39 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:41:39 dc2oss01 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 06:41:39 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394019699 last 1394019699 Mar 5 06:41:39 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 06:41:59 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST005f_UUID' is not available for connect (no target) Mar 5 06:41:59 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:42:17 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 06:42:17 dc2oss14 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 06:42:19 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 06:42:19 dc2mds01 kernel: : Lustre: Skipped 40 previous similar messages Mar 5 06:42:19 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394019739 last 1394019739 Mar 5 06:42:19 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 40 previous similar messages Mar 5 06:42:58 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:42:58 dc2oss14 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 06:43:09 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:43:09 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 06:43:33 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0095_UUID' is not available for connect (no target) Mar 5 06:43:33 dc2oss16 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:43:40 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:43:40 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 06:43:49 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:43:49 dc2oss15 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 06:44:10 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0040_UUID' is not available for connect (no target) Mar 5 06:44:10 dc2oss08 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 06:44:16 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 06:44:16 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 06:44:34 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:44:34 dc2oss06 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 06:44:30 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:44:30 dc2oss12 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 06:44:22 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:44:22 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 06:44:42 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 06:44:42 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:45:24 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:45:24 dc2oss05 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 06:45:39 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:45:39 dc2oss10 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 06:45:38 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 06:45:38 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 06:46:13 dc2oss12 kernel: : Lustre: dc2-OST0076: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 06:46:14 dc2oss12 kernel: : LustreError: 9348:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc0908800 x1461679862757550/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020006 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:14 dc2oss12 kernel: : LustreError: 9348:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 06:46:14 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:14 dc2oss12 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 06:46:23 dc2oss08 kernel: : Lustre: dc2-OST004a: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 06:46:23 dc2oss08 kernel: : LustreError: 13783:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800a4420400 x1461679862757507/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020025 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:23 dc2oss08 kernel: : LustreError: 13783:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 06:46:23 dc2oss08 kernel: : Lustre: dc2-OST004a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:23 dc2oss08 kernel: : LustreError: 27687:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88078bb60800 x1461679862757501/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020024 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:23 dc2oss08 kernel: : Lustre: dc2-OST004a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:27 dc2oss08 kernel: : Lustre: dc2-OST004e: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 06:46:27 dc2oss08 kernel: : LustreError: 14649:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88053a38fc00 x1461679862757500/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020024 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:27 dc2oss08 kernel: : LustreError: 14649:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:46:27 dc2oss08 kernel: : Lustre: dc2-OST004e: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:27 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:46:27 dc2oss10 kernel: : Lustre: dc2-OST0061: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 06:46:28 dc2oss10 kernel: : LustreError: 8801:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd8f73800 x1461679862757567/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394020004 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:28 dc2oss10 kernel: : LustreError: 8801:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 06:46:28 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:28 dc2oss10 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 06:46:42 dc2oss11 kernel: : Lustre: dc2-OST0071: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 06:46:43 dc2oss11 kernel: : LustreError: 8755:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88010ff5fc00 x1461679862757515/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020022 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:46:43 dc2oss11 kernel: : LustreError: 8755:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:46:43 dc2oss11 kernel: : Lustre: dc2-OST0071: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:46:43 dc2oss11 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 06:47:09 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 06:47:09 dc2oss12 kernel: : LustreError: Skipped 47 previous similar messages Mar 5 06:48:03 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:48:03 dc2oss08 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 06:48:03 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394020083 last 1394020083 Mar 5 06:48:03 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 06:48:15 dc2oss09 kernel: : Lustre: 16207:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394020095 last 1394020095 Mar 5 06:48:15 dc2oss09 kernel: : Lustre: 16207:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 06:48:17 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 06:48:17 dc2oss07 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 06:48:30 dc2oss11 kernel: : Lustre: dc2-OST0071: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 06:48:30 dc2oss11 kernel: : LustreError: 14694:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bee6a6800 x1461679862759396/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020201 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:48:30 dc2oss11 kernel: : Lustre: dc2-OST0071: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:48:31 dc2oss11 kernel: : LustreError: 25337:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fafe7050 x1461679862759395/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020201 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:48:31 dc2oss11 kernel: : LustreError: 25337:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 06:48:31 dc2oss11 kernel: : Lustre: dc2-OST0071: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:48:31 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 06:48:30 dc2oss08 kernel: : Lustre: dc2-OST0051: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 06:48:30 dc2oss08 kernel: : LustreError: 32192:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880295f0c800 x1461679862758582/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394020171 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 06:48:30 dc2oss08 kernel: : LustreError: 32192:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 06:48:30 dc2oss08 kernel: : Lustre: dc2-OST0051: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 06:48:30 dc2oss08 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 06:48:39 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 06:48:39 dc2oss05 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 06:49:09 dc2oss03 kernel: : Lustre: Service thread pid 21013 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 06:49:09 dc2oss03 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 06:49:09 dc2oss03 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 06:49:09 dc2oss03 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 06:49:09 dc2oss03 kernel: : LustreError: dumping log to /tmp/lustre-log.1394020149.21013 Mar 5 06:48:56 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394020136 last 1394020136 Mar 5 06:48:56 dc2oss13 kernel: : Lustre: 14522:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 06:48:56 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 06:48:56 dc2oss06 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 06:48:57 dc2oss12 kernel: : Lustre: 15383:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394020137 last 1394020137 Mar 5 06:48:57 dc2oss12 kernel: : Lustre: 15383:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 06:49:05 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394020145 last 1394020145 Mar 5 06:49:05 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 06:49:23 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 06:49:23 dc2oss15 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:49:52 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 06:49:52 dc2oss02 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:49:46 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 06:49:46 dc2oss13 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 06:49:57 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394020197 last 1394020197 Mar 5 06:49:57 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 06:50:17 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 06:50:17 dc2oss03 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 06:50:08 dc2oss15 kernel: : Lustre: 15686:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394020208 last 1394020208 Mar 5 06:50:08 dc2oss15 kernel: : Lustre: 15686:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 135 previous similar messages Mar 5 06:50:28 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:50:28 dc2oss03 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 06:50:28 dc2oss03 kernel: : Lustre: 17098:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394020228 last 1394020228 Mar 5 06:50:28 dc2oss03 kernel: : Lustre: 17098:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 06:50:21 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:50:21 dc2oss02 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 06:50:21 dc2oss02 kernel: : Lustre: 16323:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394020221 last 1394020221 Mar 5 06:50:21 dc2oss02 kernel: : Lustre: 16323:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 06:50:16 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394020216 last 1394020216 Mar 5 06:50:16 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 06:50:14 dc2oss08 kernel: : Lustre: Service thread pid 13797 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : LustreError: dumping log to /tmp/lustre-log.1394020214.13797 Mar 5 06:50:14 dc2oss08 kernel: : Lustre: Service thread pid 32322 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 06:50:14 dc2oss08 kernel: : LustreError: dumping log to /tmp/lustre-log.1394020214.32322 Mar 5 06:50:17 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394020217 last 1394020217 Mar 5 06:50:17 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 06:50:18 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394020218 last 1394020218 Mar 5 06:50:18 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 06:50:26 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 06:50:26 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:50:34 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:50:34 dc2oss04 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 06:50:46 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394020246 last 1394020246 Mar 5 06:50:46 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 06:50:55 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394020255 last 1394020255 Mar 5 06:50:55 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 182 previous similar messages Mar 5 06:51:21 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:51:21 dc2oss16 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 06:51:39 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:51:39 dc2oss01 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 06:51:39 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394020299 last 1394020299 Mar 5 06:51:39 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 06:51:50 dc2oss11 kernel: : Lustre: dc2-OST0072: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:51:50 dc2oss11 kernel: : Lustre: Skipped 181 previous similar messages Mar 5 06:52:00 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:52:00 dc2oss07 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 06:52:00 dc2oss07 kernel: : Lustre: 16000:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394020320 last 1394020320 Mar 5 06:52:00 dc2oss07 kernel: : Lustre: 16000:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 06:52:19 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0062_UUID' is not available for connect (no target) Mar 5 06:52:19 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 06:52:33 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 06:52:33 dc2oss14 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 06:52:34 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 06:52:34 dc2mds01 kernel: : Lustre: Skipped 41 previous similar messages Mar 5 06:52:34 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394020354 last 1394020354 Mar 5 06:52:34 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 41 previous similar messages Mar 5 06:53:00 dc2oss14 kernel: : Lustre: dc2-OST008d: Client a9b1d0cd-a7bc-76fe-43fb-1c0fab9149bd (at 149.165.226.202@tcp) reconnecting Mar 5 06:53:00 dc2oss14 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 06:53:13 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:53:13 dc2oss09 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 06:53:49 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:53:49 dc2oss13 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 06:53:49 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 06:53:49 dc2oss16 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 06:53:49 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:53:49 dc2oss15 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 06:53:40 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232040083975, new: 232040083974 replay: 0. see LU-617. Mar 5 06:54:14 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 06:54:14 dc2oss08 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 06:54:07 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f954:0x189:0x0]] [valid 68719476736] Mar 5 06:54:07 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 7 previous similar messages Mar 5 06:54:35 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:54:35 dc2oss06 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 06:54:24 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 06:54:24 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 06:54:49 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 06:54:49 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 06:54:58 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:54:58 dc2oss12 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 06:55:27 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:55:27 dc2oss05 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 06:55:22 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232040278842, new: 232040278841 replay: 0. see LU-617. Mar 5 06:55:43 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:55:43 dc2oss10 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 06:55:41 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 06:55:41 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 06:56:25 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 06:56:25 dc2oss11 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 06:57:10 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 06:57:10 dc2oss12 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 06:57:03 dc2mds01 kernel: : LustreError: 9827:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232040416798, new: 232040416797 replay: 0. see LU-617. Mar 5 06:58:11 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 06:58:11 dc2oss08 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 06:58:11 dc2oss08 kernel: : Lustre: 15898:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394020691 last 1394020691 Mar 5 06:58:11 dc2oss08 kernel: : Lustre: 15898:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 06:58:22 dc2oss09 kernel: : Lustre: 16472:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394020702 last 1394020702 Mar 5 06:58:22 dc2oss09 kernel: : Lustre: 16472:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 06:58:30 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 06:58:30 dc2oss07 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 06:58:42 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 06:58:42 dc2oss05 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 06:59:03 dc2oss13 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1394020743 last 1394020743 Mar 5 06:59:03 dc2oss13 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 06:59:09 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002a_UUID' is not available for connect (no target) Mar 5 06:59:09 dc2oss06 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 06:59:11 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394020751 last 1394020751 Mar 5 06:59:11 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 06:59:09 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0022: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cdc000 cur 1394020749 last 1394020749 Mar 5 06:59:09 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 06:59:32 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 06:59:32 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 06:59:55 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 06:59:55 dc2oss02 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 06:59:51 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 06:59:51 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:00:00 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd1db0000 cur 1394020800 last 1394020800 Mar 5 07:00:00 dc2oss16 kernel: : Lustre: 15771:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:00:13 dc2oss08 kernel: : Lustre: 32162:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-199), not sending early reply Mar 5 07:00:18 dc2oss08 kernel: : LustreError: 13797:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 804+0s req@ffff880bd3213800 x1461679862759154/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 3 to 0 dl 1394020818 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:00:18 dc2oss08 kernel: : Lustre: dc2-OST004a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 07:00:18 dc2oss08 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 07:00:18 dc2oss08 kernel: : Lustre: Service thread pid 13797 completed after 804.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 07:00:18 dc2oss08 kernel: : LustreError: 32322:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 804+0s req@ffff880d00a08850 x1461679862759158/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 3 to 0 dl 1394020818 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:00:18 dc2oss08 kernel: : Lustre: Service thread pid 32322 completed after 804.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 07:00:19 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394020819 last 1394020819 Mar 5 07:00:19 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 07:00:14 dc2oss15 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394020814 last 1394020814 Mar 5 07:00:14 dc2oss15 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 07:00:34 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:00:34 dc2oss03 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 07:00:34 dc2oss03 kernel: : Lustre: 17443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394020834 last 1394020834 Mar 5 07:00:34 dc2oss03 kernel: : Lustre: 17443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 07:00:29 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:00:29 dc2oss02 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 07:00:29 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394020829 last 1394020829 Mar 5 07:00:29 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 07:00:24 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394020824 last 1394020824 Mar 5 07:00:24 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 07:00:27 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394020827 last 1394020827 Mar 5 07:00:27 dc2oss14 kernel: : Lustre: 15513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 07:00:40 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:00:40 dc2oss04 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 07:00:50 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0030: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be2951800 cur 1394020850 last 1394020850 Mar 5 07:00:50 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:00:56 dc2oss11 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394020856 last 1394020856 Mar 5 07:00:56 dc2oss11 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 07:01:17 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 07:01:17 dc2oss03 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 07:01:24 dc2oss03 kernel: : Lustre: 20889:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-335), not sending early reply Mar 5 07:01:29 dc2oss03 kernel: : LustreError: 21013:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 940+0s req@ffff8806b729ac00 x1461679862757512/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 4 to 0 dl 1394020889 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:01:29 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 07:01:29 dc2oss03 kernel: : Lustre: Service thread pid 21013 completed after 940.09s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 07:01:05 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 07:01:05 dc2oss01 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 07:01:25 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:01:25 dc2oss16 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 07:02:03 dc2oss11 kernel: : Lustre: dc2-OST0072: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:02:03 dc2oss11 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:02:00 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:02:00 dc2oss01 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:02:00 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394020920 last 1394020920 Mar 5 07:02:00 dc2oss01 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 07:02:30 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:02:30 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 07:02:30 dc2oss07 kernel: : Lustre: 14316:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394020950 last 1394020950 Mar 5 07:02:30 dc2oss07 kernel: : Lustre: 14316:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 07:02:33 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 07:02:33 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 07:02:46 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:02:46 dc2mds01 kernel: : Lustre: Skipped 24 previous similar messages Mar 5 07:02:46 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394020966 last 1394020966 Mar 5 07:02:46 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 24 previous similar messages Mar 5 07:03:04 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:03:04 dc2oss14 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:03:04 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 07:03:04 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:03:29 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:03:29 dc2oss09 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 07:03:50 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:03:50 dc2oss13 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 07:03:57 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:03:57 dc2oss15 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 07:04:16 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 07:04:16 dc2oss16 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 07:04:25 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 07:04:25 dc2oss08 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 07:04:19 dc2mds01 kernel: : Lustre: 14542:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x272:0x0]] [valid 68719476736] Mar 5 07:04:19 dc2mds01 kernel: : Lustre: 14542:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 07:04:39 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:04:39 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 07:04:52 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 07:04:52 dc2oss10 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:05:02 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:05:02 dc2oss12 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 07:05:09 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:05:09 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 07:05:30 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:05:30 dc2oss05 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:05:53 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:05:53 dc2oss10 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 07:06:54 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 07:06:54 dc2oss04 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 07:07:10 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 07:07:10 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:07:19 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 07:07:19 dc2oss12 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 07:08:26 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394021306 last 1394021306 Mar 5 07:08:26 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 07:08:39 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:08:39 dc2oss08 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 07:08:39 dc2oss08 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394021319 last 1394021319 Mar 5 07:08:39 dc2oss08 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 07:08:52 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 07:08:52 dc2oss05 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 07:09:04 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394021344 last 1394021344 Mar 5 07:09:04 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 07:09:07 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 07:09:07 dc2oss07 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 07:09:09 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 07:09:09 dc2oss06 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 07:09:11 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394021351 last 1394021351 Mar 5 07:09:11 dc2oss12 kernel: : Lustre: 15291:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:09:09 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394021349 last 1394021349 Mar 5 07:09:09 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:09:10 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232041788367, new: 232041788366 replay: 0. see LU-617. Mar 5 07:09:37 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 07:09:37 dc2oss15 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:09:51 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 07:09:51 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 07:10:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 07:10:08 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 07:10:11 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394021411 last 1394021411 Mar 5 07:10:11 dc2oss16 kernel: : Lustre: 15792:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:10:22 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394021422 last 1394021422 Mar 5 07:10:22 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 07:10:33 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:10:33 dc2oss02 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:10:33 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394021433 last 1394021433 Mar 5 07:10:33 dc2oss02 kernel: : Lustre: 16362:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 07:10:45 dc2oss03 kernel: : Lustre: dc2-OST001a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:10:45 dc2oss03 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 07:10:45 dc2oss03 kernel: : Lustre: 16976:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c03192800 cur 1394021445 last 1394021445 Mar 5 07:10:45 dc2oss03 kernel: : Lustre: 16976:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 07:10:27 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394021427 last 1394021427 Mar 5 07:10:27 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 07:10:29 dc2oss14 kernel: : Lustre: 15506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1394021429 last 1394021429 Mar 5 07:10:29 dc2oss14 kernel: : Lustre: 15506:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 07:10:47 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394021447 last 1394021447 Mar 5 07:10:47 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 07:10:44 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:10:44 dc2oss04 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 07:10:53 dc2oss05 kernel: : Lustre: 16176:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394021453 last 1394021453 Mar 5 07:10:53 dc2oss05 kernel: : Lustre: 16176:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 07:10:58 dc2oss11 kernel: : Lustre: 16319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394021458 last 1394021458 Mar 5 07:10:58 dc2oss11 kernel: : Lustre: 16319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 07:11:14 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 07:11:14 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:11:45 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 07:11:45 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:11:38 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:11:38 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:11:38 dc2oss16 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 07:11:38 dc2oss16 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 07:12:03 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:12:03 dc2oss11 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 07:12:01 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:12:01 dc2oss01 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 07:12:01 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394021521 last 1394021521 Mar 5 07:12:01 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 07:12:35 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:12:35 dc2oss07 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 07:12:35 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394021555 last 1394021555 Mar 5 07:12:35 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 07:12:47 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0063_UUID' is not available for connect (no target) Mar 5 07:12:47 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:13:07 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:13:07 dc2oss14 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 07:13:08 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 07:13:08 dc2oss14 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:13:09 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 07:13:09 dc2mds01 kernel: : Lustre: Skipped 33 previous similar messages Mar 5 07:13:09 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394021589 last 1394021589 Mar 5 07:13:09 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 33 previous similar messages Mar 5 07:13:17 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232042200598, new: 232042200597 replay: 0. see LU-617. Mar 5 07:13:34 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:13:34 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 07:13:51 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:13:51 dc2oss13 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 07:14:00 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:14:00 dc2oss15 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 07:14:20 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x2a5:0x0]] [valid 68719476736] Mar 5 07:14:20 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 13 previous similar messages Mar 5 07:14:37 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 07:14:37 dc2oss08 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 07:14:45 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:14:45 dc2oss06 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:14:40 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232042312622, new: 232042312621 replay: 0. see LU-617. Mar 5 07:15:04 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 07:15:04 dc2oss16 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 07:15:13 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 07:15:13 dc2oss10 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:15:14 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:15:14 dc2oss12 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 07:15:14 dc2mds01 kernel: : LustreError: 27966:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:15:14 dc2mds01 kernel: : LustreError: 27966:0:(obd_class.h:503:obd_set_info_async()) Skipped 17 previous similar messages Mar 5 07:15:30 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:15:30 dc2oss05 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:15:56 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:15:56 dc2oss10 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 07:17:04 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0016_UUID' is not available for connect (no target) Mar 5 07:17:04 dc2oss04 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 07:17:24 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 07:17:24 dc2oss11 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:17:29 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 07:17:29 dc2oss12 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 07:18:26 dc2oss09 kernel: : Lustre: 16311:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394021906 last 1394021906 Mar 5 07:18:26 dc2oss09 kernel: : Lustre: 16311:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 07:18:42 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:18:42 dc2oss08 kernel: : Lustre: Skipped 151 previous similar messages Mar 5 07:18:42 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394021922 last 1394021922 Mar 5 07:18:42 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 151 previous similar messages Mar 5 07:19:09 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394021949 last 1394021949 Mar 5 07:19:09 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 07:19:22 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 07:19:22 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 07:19:16 dc2oss12 kernel: : Lustre: 16158:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394021956 last 1394021956 Mar 5 07:19:16 dc2oss12 kernel: : Lustre: 16158:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 07:19:11 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394021951 last 1394021951 Mar 5 07:19:11 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 07:19:46 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a4_UUID' is not available for connect (no target) Mar 5 07:19:46 dc2oss15 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 07:19:51 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 07:19:51 dc2oss06 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:20:03 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 07:20:03 dc2oss13 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 07:20:14 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818079be800 cur 1394022014 last 1394022014 Mar 5 07:20:14 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 07:20:15 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 07:20:15 dc2oss07 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 07:20:24 dc2oss15 kernel: : Lustre: 15948:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0099: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf5c5dc00 cur 1394022024 last 1394022024 Mar 5 07:20:24 dc2oss15 kernel: : Lustre: 15948:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 07:20:29 dc2oss14 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394022029 last 1394022029 Mar 5 07:20:29 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394022029 last 1394022029 Mar 5 07:20:29 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 07:20:49 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:20:49 dc2oss03 kernel: : Lustre: Skipped 176 previous similar messages Mar 5 07:20:49 dc2oss03 kernel: : Lustre: 16980:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394022049 last 1394022049 Mar 5 07:20:49 dc2oss03 kernel: : Lustre: 16980:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 176 previous similar messages Mar 5 07:20:43 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394022043 last 1394022043 Mar 5 07:20:43 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:20:51 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394022051 last 1394022051 Mar 5 07:20:51 dc2oss10 kernel: : Lustre: 16459:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 07:20:50 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:20:50 dc2oss04 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 07:20:55 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0030: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be2951800 cur 1394022055 last 1394022055 Mar 5 07:20:55 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 07:21:06 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:21:06 dc2oss02 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 07:21:06 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394022066 last 1394022066 Mar 5 07:21:06 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 07:21:08 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394022068 last 1394022068 Mar 5 07:21:08 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 07:21:49 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 07:21:49 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:21:39 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:21:39 dc2oss16 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 07:21:36 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 07:21:36 dc2oss01 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 07:22:04 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:22:04 dc2oss11 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 07:22:01 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:22:01 dc2oss01 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 07:22:01 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394022121 last 1394022121 Mar 5 07:22:01 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 07:22:44 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 07:22:44 dc2oss02 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:22:35 dc2oss07 kernel: : Lustre: dc2-OST0044: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:22:35 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 07:22:35 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12508400 cur 1394022155 last 1394022155 Mar 5 07:22:35 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 07:22:29 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232043164971, new: 232043164970 replay: 0. see LU-617. Mar 5 07:23:01 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0067_UUID' is not available for connect (no target) Mar 5 07:23:01 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 07:23:07 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:23:07 dc2oss14 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:23:09 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 07:23:09 dc2mds01 kernel: : Lustre: Skipped 15 previous similar messages Mar 5 07:23:09 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394022189 last 1394022189 Mar 5 07:23:09 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 15 previous similar messages Mar 5 07:23:23 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 07:23:23 dc2oss14 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:23:42 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:23:42 dc2oss09 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 07:24:01 dc2oss15 kernel: : Lustre: dc2-OST0097: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:24:01 dc2oss15 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 07:24:02 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:24:02 dc2oss13 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 07:24:41 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 07:24:41 dc2oss08 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:25:03 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:25:03 dc2oss06 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 07:25:07 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0099_UUID' is not available for connect (no target) Mar 5 07:25:07 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 07:25:18 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 07:25:18 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 07:25:43 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:25:43 dc2oss12 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 07:25:52 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:25:52 dc2oss05 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 07:26:00 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:26:00 dc2oss10 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 07:26:28 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:26:28 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 11 previous similar messages Mar 5 07:27:36 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001e_UUID' is not available for connect (no target) Mar 5 07:27:36 dc2oss04 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:28:10 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 07:28:10 dc2oss11 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 07:28:30 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394022510 last 1394022510 Mar 5 07:28:30 dc2oss09 kernel: : Lustre: 16163:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 07:28:45 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:28:45 dc2oss08 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 07:28:45 dc2oss08 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394022525 last 1394022525 Mar 5 07:28:45 dc2oss08 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 07:29:23 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 07:29:23 dc2oss05 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 07:29:15 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394022555 last 1394022555 Mar 5 07:29:15 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 07:29:16 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394022556 last 1394022556 Mar 5 07:29:16 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 07:29:19 dc2oss04 kernel: : Lustre: 16419:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394022559 last 1394022559 Mar 5 07:29:19 dc2oss04 kernel: : Lustre: 16419:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 07:29:27 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232043907815, new: 232043907814 replay: 0. see LU-617. Mar 5 07:30:07 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 07:30:07 dc2oss15 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 07:30:11 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 07:30:11 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 07:30:16 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 07:30:16 dc2oss12 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 07:30:24 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 07:30:24 dc2oss07 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:30:27 dc2oss16 kernel: : Lustre: 17221:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394022627 last 1394022627 Mar 5 07:30:27 dc2oss16 kernel: : Lustre: 17221:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 07:30:29 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0097: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf83cf000 cur 1394022629 last 1394022629 Mar 5 07:30:29 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 07:30:50 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:30:50 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 07:30:50 dc2oss03 kernel: : Lustre: 17108:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394022650 last 1394022650 Mar 5 07:30:50 dc2oss03 kernel: : Lustre: 17108:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 07:30:32 dc2oss14 kernel: : Lustre: 15670:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394022632 last 1394022632 Mar 5 07:30:32 dc2oss14 kernel: : Lustre: 15670:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 07:30:44 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394022644 last 1394022644 Mar 5 07:30:44 dc2oss06 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 07:30:47 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 07:30:47 dc2oss13 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 07:30:54 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394022654 last 1394022654 Mar 5 07:30:54 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:30:53 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:30:53 dc2oss04 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 07:31:00 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394022660 last 1394022660 Mar 5 07:31:00 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 07:31:18 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:31:18 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 07:31:18 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394022678 last 1394022678 Mar 5 07:31:18 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 07:31:10 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394022670 last 1394022670 Mar 5 07:31:10 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 07:31:40 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:31:40 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 07:32:11 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:32:11 dc2oss11 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:32:14 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:32:14 dc2oss01 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 07:32:14 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394022734 last 1394022734 Mar 5 07:32:14 dc2oss01 kernel: : Lustre: 15338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 07:32:21 dc2mds01 kernel: : Lustre: 1368:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x2b5:0x0]] [valid 68719476736] Mar 5 07:32:21 dc2mds01 kernel: : Lustre: 1368:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 3 previous similar messages Mar 5 07:32:26 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 07:32:26 dc2oss01 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:32:38 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:32:38 dc2oss07 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 07:32:38 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394022758 last 1394022758 Mar 5 07:32:38 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 07:33:05 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 07:33:05 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 07:33:15 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 07:33:15 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:33:27 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 07:33:27 dc2oss03 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 07:33:14 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:33:14 dc2oss14 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 07:33:27 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 07:33:27 dc2oss14 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 07:33:20 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 07:33:20 dc2mds01 kernel: : Lustre: Skipped 17 previous similar messages Mar 5 07:33:20 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394022800 last 1394022800 Mar 5 07:33:20 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 17 previous similar messages Mar 5 07:33:47 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:33:47 dc2oss09 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 07:34:01 dc2oss15 kernel: : Lustre: dc2-OST0095: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:34:01 dc2oss15 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 07:34:07 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:34:07 dc2oss13 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 07:34:47 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 07:34:47 dc2oss08 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 07:35:08 dc2oss06 kernel: : Lustre: dc2-OST003e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:35:08 dc2oss06 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 07:35:31 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 07:35:31 dc2oss16 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 07:35:33 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 07:35:33 dc2oss10 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 07:35:53 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:35:53 dc2oss12 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:35:56 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:35:56 dc2oss05 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:36:04 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:36:04 dc2oss10 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 07:36:32 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:36:32 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 12 previous similar messages Mar 5 07:37:23 dc2oss12 kernel: : Lustre: dc2-OST0076: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 07:37:23 dc2oss12 kernel: : LustreError: 9335:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88007be5f800 x1461679862822531/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394023087 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:37:23 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 07:37:23 dc2oss12 kernel: : LustreError: 32688:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803f5050800 x1461679862822534/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394023087 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:37:23 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 07:37:44 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 07:37:44 dc2oss04 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 07:38:17 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 07:38:17 dc2oss11 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 07:38:30 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394023110 last 1394023110 Mar 5 07:38:30 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 07:38:45 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:38:45 dc2oss08 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:38:45 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394023125 last 1394023125 Mar 5 07:38:45 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:39:21 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394023161 last 1394023161 Mar 5 07:39:21 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 07:39:23 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff8813760e9000 cur 1394023163 last 1394023163 Mar 5 07:39:23 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 07:39:19 dc2oss04 kernel: : Lustre: 16099:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0022: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cdc000 cur 1394023159 last 1394023159 Mar 5 07:39:19 dc2oss04 kernel: : Lustre: 16099:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 07:39:24 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 07:39:24 dc2oss04 kernel: : LustreError: 16239:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88076858cc00 x1461679898669843/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 0 to 0 dl 1394023173 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:39:24 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 07:39:24 dc2oss04 kernel: : LustreError: 20053:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b085a7800 x1461679898669818/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 0 to 0 dl 1394023170 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:39:24 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 07:39:25 dc2oss04 kernel: : LustreError: 19860:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88001fee0c00 x1461679898669804/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394023231 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:39:25 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 07:39:34 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 07:39:34 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 07:40:29 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394023229 last 1394023229 Mar 5 07:40:29 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 07:40:28 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004e_UUID' is not available for connect (no target) Mar 5 07:40:28 dc2oss07 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:40:35 dc2oss14 kernel: : Lustre: 15782:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394023235 last 1394023235 Mar 5 07:40:35 dc2oss14 kernel: : Lustre: 15782:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:40:55 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:40:55 dc2oss03 kernel: : Lustre: Skipped 166 previous similar messages Mar 5 07:40:55 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394023255 last 1394023255 Mar 5 07:40:55 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 167 previous similar messages Mar 5 07:40:35 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394023235 last 1394023235 Mar 5 07:40:35 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:40:38 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 07:40:38 dc2oss15 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:40:47 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394023247 last 1394023247 Mar 5 07:40:47 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 07:40:48 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 07:40:48 dc2oss12 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:40:54 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394023254 last 1394023254 Mar 5 07:40:54 dc2oss10 kernel: : Lustre: 16313:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 07:40:59 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:40:59 dc2oss04 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 07:41:06 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 07:41:06 dc2oss06 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 07:41:09 dc2oss05 kernel: : Lustre: 15933:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394023269 last 1394023269 Mar 5 07:41:09 dc2oss05 kernel: : Lustre: 15933:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 07:41:10 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394023270 last 1394023270 Mar 5 07:41:10 dc2oss11 kernel: : Lustre: 16209:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 185 previous similar messages Mar 5 07:41:26 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:41:26 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 07:41:26 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394023286 last 1394023286 Mar 5 07:41:26 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 07:41:28 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 07:41:28 dc2oss13 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:41:45 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:41:45 dc2oss16 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 07:42:19 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:42:19 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:42:19 dc2oss11 kernel: : Lustre: Skipped 186 previous similar messages Mar 5 07:42:27 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:42:27 dc2oss01 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:42:27 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394023347 last 1394023347 Mar 5 07:42:27 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 07:42:31 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 07:42:31 dc2oss01 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 07:42:39 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:42:39 dc2oss07 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 07:42:39 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394023359 last 1394023359 Mar 5 07:42:39 dc2oss07 kernel: : Lustre: 15810:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 07:43:23 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 07:43:23 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:43:24 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:43:24 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:43:24 dc2oss14 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:43:24 dc2oss14 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 07:43:50 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 07:43:50 dc2oss03 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 07:43:21 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 07:43:21 dc2mds01 kernel: : Lustre: Skipped 35 previous similar messages Mar 5 07:43:21 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394023401 last 1394023401 Mar 5 07:43:21 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 35 previous similar messages Mar 5 07:43:33 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0067_UUID' is not available for connect (no target) Mar 5 07:43:33 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:43:41 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0080_UUID' is not available for connect (no target) Mar 5 07:43:41 dc2oss14 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 07:43:49 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:43:49 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 07:44:05 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:44:05 dc2oss15 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 07:44:11 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:44:11 dc2oss13 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:44:32 dc2mds01 kernel: : Lustre: 1369:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f959:0x9f:0x0]] [valid 68719476736] Mar 5 07:44:32 dc2mds01 kernel: : Lustre: 1369:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 07:44:55 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 07:44:55 dc2oss08 kernel: : LustreError: Skipped 40 previous similar messages Mar 5 07:45:11 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:45:11 dc2oss06 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 07:45:31 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 07:45:31 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 07:45:54 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:45:54 dc2oss12 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 07:45:59 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:45:59 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 07:46:14 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:46:14 dc2oss10 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 07:46:37 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 07:46:37 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 07:47:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:47:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 07:48:14 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 07:48:14 dc2oss04 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 07:48:28 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 07:48:28 dc2oss11 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 07:48:38 dc2oss09 kernel: : Lustre: 16431:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394023718 last 1394023718 Mar 5 07:48:38 dc2oss09 kernel: : Lustre: 16431:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 07:48:46 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:48:46 dc2oss08 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:48:46 dc2oss08 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394023726 last 1394023726 Mar 5 07:48:46 dc2oss08 kernel: : Lustre: 16148:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 07:48:54 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232046057322, new: 232046057321 replay: 0. see LU-617. Mar 5 07:49:23 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394023763 last 1394023763 Mar 5 07:49:23 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 07:49:34 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394023774 last 1394023774 Mar 5 07:49:34 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 07:49:31 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394023771 last 1394023771 Mar 5 07:49:31 dc2oss04 kernel: : Lustre: 16768:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 07:49:49 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 07:49:49 dc2oss05 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 07:50:37 dc2oss15 kernel: : Lustre: 16348:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394023837 last 1394023837 Mar 5 07:50:37 dc2oss15 kernel: : Lustre: 16348:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 07:50:56 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:50:56 dc2oss03 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 07:50:56 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394023856 last 1394023856 Mar 5 07:50:56 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 07:50:39 dc2oss16 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05c45400 cur 1394023839 last 1394023839 Mar 5 07:50:39 dc2oss16 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 07:50:44 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394023844 last 1394023844 Mar 5 07:50:44 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 07:50:47 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 07:50:47 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:50:47 dc2oss06 kernel: : Lustre: 14430:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394023847 last 1394023847 Mar 5 07:50:47 dc2oss06 kernel: : Lustre: 14430:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 07:50:48 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 07:50:48 dc2oss12 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 07:50:54 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 07:50:54 dc2oss07 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 07:50:58 dc2oss10 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1394023858 last 1394023858 Mar 5 07:50:58 dc2oss10 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 07:51:15 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394023875 last 1394023875 Mar 5 07:51:15 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 07:51:15 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394023875 last 1394023875 Mar 5 07:51:15 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 07:51:18 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:51:18 dc2oss04 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 07:51:35 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:51:35 dc2oss02 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 07:51:35 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394023895 last 1394023895 Mar 5 07:51:35 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 07:51:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 07:51:33 dc2oss13 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 07:51:36 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 07:51:36 dc2oss06 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 07:52:03 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:52:03 dc2oss16 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 07:52:21 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:52:21 dc2oss11 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 07:52:46 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:52:46 dc2oss01 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 07:52:46 dc2oss01 kernel: : Lustre: 15377:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394023966 last 1394023966 Mar 5 07:52:46 dc2oss01 kernel: : Lustre: 15377:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 86 previous similar messages Mar 5 07:53:08 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 07:53:08 dc2oss01 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 07:53:19 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:53:19 dc2oss07 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 07:53:19 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394023999 last 1394023999 Mar 5 07:53:19 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 07:53:36 dc2oss02 kernel: : Lustre: dc2-OST000c: Client 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 07:53:36 dc2oss02 kernel: : LustreError: 15983:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8807922fb400 x1461680408341028/t0(0) o4->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/416 e 1 to 0 dl 1394024064 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:53:36 dc2oss02 kernel: : LustreError: 15983:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 07:53:36 dc2oss02 kernel: : Lustre: dc2-OST000c: Bulk IO write error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 07:53:36 dc2oss02 kernel: : LustreError: 20652:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff88008b3a7800 x1461680408340768/t0(0) o4->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/416 e 1 to 0 dl 1394024058 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 07:53:36 dc2oss02 kernel: : Lustre: dc2-OST000c: Bulk IO write error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 07:53:26 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:53:26 dc2oss14 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 07:53:38 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 07:53:38 dc2oss09 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 07:53:53 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 07:53:53 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 07:53:58 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:53:58 dc2oss09 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:54:00 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 07:54:00 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:53:58 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232046583347, new: 232046583346 replay: 0. see LU-617. Mar 5 07:54:12 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:54:12 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 07:54:14 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:54:14 dc2oss13 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 07:54:23 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 07:54:23 dc2mds01 kernel: : Lustre: Skipped 34 previous similar messages Mar 5 07:54:23 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394024063 last 1394024063 Mar 5 07:54:23 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 34 previous similar messages Mar 5 07:55:11 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:55:11 dc2oss06 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 07:55:17 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0044_UUID' is not available for connect (no target) Mar 5 07:55:17 dc2oss08 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 07:55:57 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 07:55:57 dc2oss16 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 07:56:04 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:56:04 dc2oss12 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 07:56:06 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:56:06 dc2oss05 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 07:56:18 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:56:18 dc2oss10 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 07:56:33 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232046860331, new: 232046860330 replay: 0. see LU-617. Mar 5 07:57:00 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 07:57:00 dc2oss03 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 07:57:12 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 07:57:12 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 07:57:51 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 07:57:51 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 33 previous similar messages Mar 5 07:58:35 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 07:58:35 dc2oss11 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 07:58:22 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 07:58:22 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 07:58:43 dc2oss09 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394024323 last 1394024323 Mar 5 07:58:43 dc2oss09 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 07:58:48 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 07:58:48 dc2oss08 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 07:58:48 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394024328 last 1394024328 Mar 5 07:58:48 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 07:59:28 dc2oss12 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394024368 last 1394024368 Mar 5 07:59:28 dc2oss12 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 07:59:37 dc2oss13 kernel: : Lustre: 15856:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394024377 last 1394024377 Mar 5 07:59:37 dc2oss13 kernel: : Lustre: 15856:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 07:59:40 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394024380 last 1394024380 Mar 5 07:59:40 dc2oss04 kernel: : Lustre: 16720:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 08:00:22 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 08:00:22 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 08:00:37 dc2oss15 kernel: : Lustre: 15919:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394024437 last 1394024437 Mar 5 08:00:37 dc2oss15 kernel: : Lustre: 15919:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 08:00:39 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05c45400 cur 1394024439 last 1394024439 Mar 5 08:00:39 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 08:00:49 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394024449 last 1394024449 Mar 5 08:00:49 dc2oss14 kernel: : Lustre: 15548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 08:00:50 dc2oss06 kernel: : Lustre: 14430:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394024450 last 1394024450 Mar 5 08:00:50 dc2oss06 kernel: : Lustre: 14430:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 08:00:52 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 08:00:52 dc2oss12 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 08:01:13 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:01:13 dc2oss03 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 08:01:13 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394024473 last 1394024473 Mar 5 08:01:13 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 08:01:21 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 08:01:21 dc2oss15 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 08:01:19 dc2oss05 kernel: : Lustre: 16176:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394024479 last 1394024479 Mar 5 08:01:19 dc2oss05 kernel: : Lustre: 16176:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 08:01:15 dc2oss10 kernel: : Lustre: 16579:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394024475 last 1394024475 Mar 5 08:01:15 dc2oss10 kernel: : Lustre: 16579:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 08:01:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394024475 last 1394024475 Mar 5 08:01:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:01:21 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:01:21 dc2oss04 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 08:01:37 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 08:01:37 dc2oss06 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:01:38 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 08:01:38 dc2oss13 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 08:01:50 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:01:50 dc2oss02 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 08:01:50 dc2oss02 kernel: : Lustre: 16224:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394024510 last 1394024510 Mar 5 08:01:50 dc2oss02 kernel: : Lustre: 16224:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 08:02:09 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:02:09 dc2oss16 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 08:02:25 dc2oss11 kernel: : Lustre: dc2-OST0073: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:02:25 dc2oss11 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 08:03:11 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 08:03:11 dc2oss07 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 08:03:20 dc2oss07 kernel: : Lustre: dc2-OST0047: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:03:20 dc2oss07 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 08:03:20 dc2oss07 kernel: : Lustre: 15965:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1394024600 last 1394024600 Mar 5 08:03:20 dc2oss07 kernel: : Lustre: 15965:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 08:03:04 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:03:04 dc2oss01 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 08:03:04 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394024584 last 1394024584 Mar 5 08:03:04 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 08:03:26 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 08:03:26 dc2oss01 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 08:03:28 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:03:28 dc2oss14 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 08:03:39 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 08:03:39 dc2oss09 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 08:04:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 08:04:08 dc2oss02 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 08:03:58 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:03:58 dc2oss09 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 08:04:14 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:04:14 dc2oss13 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 08:04:20 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:04:20 dc2oss15 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:04:27 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 08:04:27 dc2mds01 kernel: : Lustre: Skipped 34 previous similar messages Mar 5 08:04:27 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394024667 last 1394024667 Mar 5 08:04:27 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 34 previous similar messages Mar 5 08:05:10 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 08:05:10 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:05:16 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:05:16 dc2oss06 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 08:05:32 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 08:05:32 dc2oss08 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 08:06:05 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 08:06:05 dc2oss16 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 08:06:11 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:06:11 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 08:06:14 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:06:14 dc2oss05 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 08:06:20 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:06:20 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 08:07:12 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 08:07:12 dc2oss03 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 08:07:46 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 08:07:46 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 08:07:55 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 08:07:55 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 18 previous similar messages Mar 5 08:08:53 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:08:53 dc2oss08 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 08:08:53 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394024933 last 1394024933 Mar 5 08:08:53 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 134 previous similar messages Mar 5 08:08:39 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f958:0x698:0x0]] [valid 68719476736] Mar 5 08:08:39 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 08:08:53 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394024933 last 1394024933 Mar 5 08:08:53 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 08:09:12 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 08:09:12 dc2oss04 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 08:09:26 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 08:09:26 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 08:09:30 dc2oss12 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0075: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0d854800 cur 1394024970 last 1394024970 Mar 5 08:09:30 dc2oss12 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 08:09:37 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1394024977 last 1394024977 Mar 5 08:09:37 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 08:09:44 dc2oss04 kernel: : Lustre: 16374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394024984 last 1394024984 Mar 5 08:09:44 dc2oss04 kernel: : Lustre: 16374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:10:40 dc2oss15 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394025040 last 1394025040 Mar 5 08:10:40 dc2oss15 kernel: : Lustre: 15851:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394025040 last 1394025040 Mar 5 08:10:40 dc2oss15 kernel: : Lustre: 15851:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 08:10:40 dc2oss15 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 08:10:49 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394025049 last 1394025049 Mar 5 08:10:49 dc2oss16 kernel: : Lustre: 17178:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 08:10:50 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1394025050 last 1394025050 Mar 5 08:10:50 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:11:13 dc2oss03 kernel: : Lustre: dc2-OST0016: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:11:13 dc2oss03 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 08:11:13 dc2oss03 kernel: : Lustre: 16915:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31136294 exp ffff880c0266f000 cur 1394025073 last 1394025073 Mar 5 08:11:13 dc2oss03 kernel: : Lustre: 16915:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 08:10:51 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394025051 last 1394025051 Mar 5 08:10:51 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 08:11:01 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 08:11:01 dc2oss12 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 08:11:28 dc2oss11 kernel: : Lustre: 15496:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394025088 last 1394025088 Mar 5 08:11:28 dc2oss11 kernel: : Lustre: 15496:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 08:11:26 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394025086 last 1394025086 Mar 5 08:11:26 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 08:11:28 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 08:11:28 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 08:11:21 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:11:21 dc2oss04 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 08:11:55 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:11:55 dc2oss02 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 08:11:55 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394025115 last 1394025115 Mar 5 08:11:55 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 08:11:42 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394025102 last 1394025102 Mar 5 08:11:42 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 08:11:59 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 08:11:59 dc2oss06 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:12:06 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 08:12:06 dc2oss13 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 08:12:10 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 08:12:10 dc2oss15 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 08:12:11 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:12:11 dc2oss16 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 08:12:30 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:12:30 dc2oss11 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 08:13:08 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:13:08 dc2oss01 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 08:13:08 dc2oss01 kernel: : Lustre: 15605:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394025188 last 1394025188 Mar 5 08:13:08 dc2oss01 kernel: : Lustre: 15605:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 08:13:23 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 08:13:23 dc2oss07 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 08:13:24 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:13:24 dc2oss07 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:13:24 dc2oss07 kernel: : Lustre: 14437:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394025204 last 1394025204 Mar 5 08:13:24 dc2oss07 kernel: : Lustre: 14437:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 08:13:36 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:13:36 dc2oss14 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 08:13:36 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 08:13:36 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:13:49 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 08:13:49 dc2oss09 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 08:13:58 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:13:58 dc2oss09 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 08:14:14 dc2oss13 kernel: : Lustre: dc2-OST0085: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:14:14 dc2oss13 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 08:14:37 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 08:14:37 dc2oss02 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 08:14:39 dc2oss15 kernel: : Lustre: dc2-OST0098: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:14:39 dc2oss15 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:14:36 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 08:14:36 dc2mds01 kernel: : Lustre: Skipped 16 previous similar messages Mar 5 08:14:36 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394025276 last 1394025276 Mar 5 08:14:36 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages Mar 5 08:15:12 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0081_UUID' is not available for connect (no target) Mar 5 08:15:12 dc2oss14 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 08:15:16 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:15:16 dc2oss06 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 08:15:32 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 08:15:32 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:16:12 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 08:16:12 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 08:16:15 dc2oss05 kernel: : Lustre: dc2-OST002a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:16:15 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 08:16:17 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:16:17 dc2oss12 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:16:34 dc2oss10 kernel: : Lustre: dc2-OST0067: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:16:34 dc2oss10 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 08:17:21 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x73d2:0x0]] [valid 68719476736] Mar 5 08:17:21 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 08:17:55 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 08:17:55 dc2oss10 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 08:18:25 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 08:18:25 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 9 previous similar messages Mar 5 08:19:11 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 08:19:11 dc2oss03 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 08:18:59 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:18:59 dc2oss08 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 08:18:59 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394025539 last 1394025539 Mar 5 08:18:59 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 08:18:55 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394025535 last 1394025535 Mar 5 08:18:55 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 08:19:00 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:01 dc2mds01 kernel: : Lustre: 11130:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:01 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:01 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) Skipped 7 previous similar messages Mar 5 08:19:02 dc2mds01 kernel: : Lustre: 14548:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:02 dc2mds01 kernel: : Lustre: 14548:0:(mdd_device.c:900:obf_lookup()) Skipped 20 previous similar messages Mar 5 08:19:05 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:05 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) Skipped 42 previous similar messages Mar 5 08:19:09 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:09 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) Skipped 50 previous similar messages Mar 5 08:19:17 dc2mds01 kernel: : Lustre: 11151:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:17 dc2mds01 kernel: : Lustre: 11151:0:(mdd_device.c:900:obf_lookup()) Skipped 94 previous similar messages Mar 5 08:19:34 dc2oss12 kernel: : Lustre: 14470:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394025574 last 1394025574 Mar 5 08:19:34 dc2oss12 kernel: : Lustre: 14470:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 08:19:40 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394025580 last 1394025580 Mar 5 08:19:40 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 08:19:33 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x201ead4c7:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 08:19:33 dc2mds01 kernel: : Lustre: 1368:0:(mdd_device.c:900:obf_lookup()) Skipped 169 previous similar messages Mar 5 08:19:42 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007d_UUID' is not available for connect (no target) Mar 5 08:19:42 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:19:45 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04cab800 cur 1394025585 last 1394025585 Mar 5 08:19:45 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 76 previous similar messages Mar 5 08:20:07 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 08:20:07 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:20:44 dc2oss15 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394025644 last 1394025644 Mar 5 08:20:44 dc2oss15 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 08:20:52 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394025652 last 1394025652 Mar 5 08:20:52 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 08:20:58 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394025658 last 1394025658 Mar 5 08:20:58 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 08:20:58 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd1db0000 cur 1394025658 last 1394025658 Mar 5 08:20:58 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 08:21:26 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:21:26 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 08:21:26 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394025686 last 1394025686 Mar 5 08:21:26 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 08:21:21 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 08:21:21 dc2oss12 kernel: : LustreError: Skipped 42 previous similar messages Mar 5 08:21:29 dc2oss05 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fae4a000 cur 1394025689 last 1394025689 Mar 5 08:21:29 dc2oss05 kernel: : Lustre: 15945:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 08:21:32 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 08:21:32 dc2oss05 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 08:21:25 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:21:25 dc2oss04 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 08:21:37 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394025697 last 1394025697 Mar 5 08:21:37 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 08:21:59 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:21:59 dc2oss02 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 08:21:59 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1394025719 last 1394025719 Mar 5 08:21:59 dc2oss02 kernel: : Lustre: 16455:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 08:21:46 dc2oss10 kernel: : Lustre: 16486:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394025706 last 1394025706 Mar 5 08:21:46 dc2oss10 kernel: : Lustre: 16486:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 08:22:07 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 08:22:07 dc2oss13 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:22:15 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:22:15 dc2oss16 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:22:22 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002a_UUID' is not available for connect (no target) Mar 5 08:22:22 dc2oss06 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:22:32 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:22:32 dc2oss11 kernel: : Lustre: Skipped 133 previous similar messages Mar 5 08:22:51 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 08:22:51 dc2oss15 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 08:23:11 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:23:11 dc2oss01 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 08:23:11 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394025791 last 1394025791 Mar 5 08:23:11 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 08:23:28 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:23:28 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 08:23:28 dc2oss07 kernel: : Lustre: 15907:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394025808 last 1394025808 Mar 5 08:23:28 dc2oss07 kernel: : Lustre: 15907:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 08:23:42 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:23:42 dc2oss14 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:23:52 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 08:23:52 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 08:23:57 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 08:23:57 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 08:24:04 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:24:04 dc2oss09 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 08:23:48 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 08:23:48 dc2oss01 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:24:14 dc2oss13 kernel: : Lustre: dc2-OST0086: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:24:14 dc2oss13 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 08:24:41 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 08:24:41 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 08:24:41 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:24:41 dc2oss15 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 08:24:50 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:24:50 dc2mds01 kernel: : Lustre: Skipped 32 previous similar messages Mar 5 08:24:50 dc2mds01 kernel: : Lustre: 11151:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394025890 last 1394025890 Mar 5 08:24:50 dc2mds01 kernel: : Lustre: 11151:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 32 previous similar messages Mar 5 08:25:16 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:25:16 dc2oss06 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 08:25:42 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 08:25:42 dc2oss08 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 08:26:08 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 08:26:08 dc2oss14 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:26:01 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f79d:0xc:0x0]] [valid 68719476736] Mar 5 08:26:01 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 3 previous similar messages Mar 5 08:26:15 dc2oss05 kernel: : Lustre: dc2-OST0033: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:26:15 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 08:26:14 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 08:26:14 dc2oss16 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 08:26:17 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:26:17 dc2oss12 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 08:26:39 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:26:39 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 08:28:49 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 08:28:49 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 08:29:21 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 08:29:21 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 08:29:01 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394026141 last 1394026141 Mar 5 08:29:01 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 08:29:13 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:29:13 dc2oss08 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 08:29:13 dc2oss08 kernel: : Lustre: 16072:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394026153 last 1394026153 Mar 5 08:29:13 dc2oss08 kernel: : Lustre: 16072:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 68 previous similar messages Mar 5 08:29:38 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394026178 last 1394026178 Mar 5 08:29:38 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 08:29:45 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394026185 last 1394026185 Mar 5 08:29:45 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 08:30:01 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394026201 last 1394026201 Mar 5 08:30:01 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 08:30:24 dc2oss12 kernel: : Lustre: dc2-OST0076: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 08:30:24 dc2oss12 kernel: : LustreError: 14612:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bcfe57800 x1461679862884555/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026257 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:24 dc2oss12 kernel: : LustreError: 14612:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 08:30:24 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:24 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:30:24 dc2oss12 kernel: : LustreError: 9243:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803f8b8c000 x1461679862884556/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026257 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:24 dc2oss12 kernel: : LustreError: 9243:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 08:30:24 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:24 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:30:13 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 08:30:13 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 21 previous similar messages Mar 5 08:30:26 dc2oss10 kernel: : Lustre: dc2-OST0061: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 08:30:26 dc2oss10 kernel: : LustreError: 15902:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8805196fe050 x1461679862884590/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026262 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:26 dc2oss10 kernel: : LustreError: 15902:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 08:30:26 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:26 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:30:27 dc2oss10 kernel: : LustreError: 8733:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd701d000 x1461679862884585/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026262 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:27 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:38 dc2oss08 kernel: : Lustre: dc2-OST0051: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 08:30:38 dc2oss08 kernel: : LustreError: 27750:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd7a61800 x1461679862884580/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026291 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:38 dc2oss08 kernel: : LustreError: 27750:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 08:30:38 dc2oss08 kernel: : Lustre: dc2-OST0051: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:38 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:30:45 dc2oss08 kernel: : Lustre: dc2-OST004e: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 08:30:46 dc2oss08 kernel: : LustreError: 14593:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880ff58d9050 x1461679862884575/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026291 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:30:46 dc2oss08 kernel: : LustreError: 14593:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 08:30:46 dc2oss08 kernel: : Lustre: dc2-OST004e: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:46 dc2oss08 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 08:30:39 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0019_UUID' is not available for connect (no target) Mar 5 08:30:39 dc2oss04 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 08:30:44 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 08:30:44 dc2oss11 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 08:31:08 dc2oss03 kernel: : Lustre: dc2-OST001a: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 08:31:08 dc2oss03 kernel: : LustreError: 16820:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806ff906400 x1461679862884567/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394026281 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:31:08 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:31:09 dc2oss03 kernel: : LustreError: 20376:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf8bc6800 x1461679862884551/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394026280 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:31:09 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:30:50 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp t0 exp ffff8813fb4e5c00 cur 1394026250 last 1394026250 Mar 5 08:30:50 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 08:30:54 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394026254 last 1394026254 Mar 5 08:30:54 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 08:31:07 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394026267 last 1394026267 Mar 5 08:31:07 dc2oss06 kernel: : Lustre: 15993:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:31:30 dc2oss03 kernel: : Lustre: dc2-OST001a: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) reconnecting Mar 5 08:31:30 dc2oss03 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 08:31:30 dc2oss03 kernel: : Lustre: dc2-OST001a: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 08:31:30 dc2oss03 kernel: : LustreError: 20897:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806b9a43800 x1461679862885819/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394026356 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:31:30 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:31:30 dc2oss03 kernel: : Lustre: 17094:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394026290 last 1394026290 Mar 5 08:31:30 dc2oss03 kernel: : Lustre: 17094:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 08:31:13 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp t0 exp ffff880c8ab5d400 cur 1394026273 last 1394026273 Mar 5 08:31:13 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 08:31:31 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394026291 last 1394026291 Mar 5 08:31:31 dc2oss05 kernel: : Lustre: 14558:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 08:31:28 dc2oss04 kernel: : Lustre: dc2-OST0022: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:31:28 dc2oss04 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:31:39 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394026299 last 1394026299 Mar 5 08:31:39 dc2oss11 kernel: : Lustre: 16184:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:31:51 dc2oss10 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394026311 last 1394026311 Mar 5 08:31:51 dc2oss10 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 08:32:08 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:32:08 dc2oss02 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 08:32:08 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394026328 last 1394026328 Mar 5 08:32:08 dc2oss02 kernel: : Lustre: 16445:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 08:32:10 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 08:32:10 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:32:12 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 08:32:12 dc2oss05 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 08:32:16 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:32:16 dc2oss16 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:32:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 08:32:33 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:32:41 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:32:41 dc2oss11 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 08:32:43 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 08:32:43 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:32:46 dc2oss08 kernel: : Lustre: dc2-OST0051: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 08:32:47 dc2oss08 kernel: : LustreError: 27728:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd4d67400 x1461679862885842/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394026458 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 08:32:47 dc2oss08 kernel: : Lustre: dc2-OST0051: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:33:11 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 08:33:11 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 08:33:20 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:33:20 dc2oss01 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 08:33:20 dc2oss01 kernel: : Lustre: 15033:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394026400 last 1394026400 Mar 5 08:33:20 dc2oss01 kernel: : Lustre: 15033:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 08:33:46 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:33:46 dc2oss07 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 08:33:46 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394026426 last 1394026426 Mar 5 08:33:46 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 129 previous similar messages Mar 5 08:33:46 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:33:46 dc2oss14 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 08:34:00 dc2oss10 kernel: : Lustre: Service thread pid 15909 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:34:00 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:34:00 dc2oss10 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 08:34:00 dc2oss10 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 08:34:00 dc2oss10 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:34:00 dc2oss10 kernel: : LustreError: dumping log to /tmp/lustre-log.1394026440.15909 Mar 5 08:34:01 dc2oss12 kernel: : Lustre: Service thread pid 32676 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:34:01 dc2oss12 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 08:34:01 dc2oss12 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 08:34:01 dc2oss12 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:34:01 dc2oss12 kernel: : LustreError: dumping log to /tmp/lustre-log.1394026441.32676 Mar 5 08:34:09 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:34:09 dc2oss09 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 08:34:04 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 08:34:04 dc2oss07 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 08:33:56 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 08:33:56 dc2oss01 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 08:34:19 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:34:19 dc2oss13 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 08:34:19 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 08:34:19 dc2oss09 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 08:34:47 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0005_UUID' is not available for connect (no target) Mar 5 08:34:47 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:34:58 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:34:58 dc2oss15 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 08:35:17 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:35:17 dc2oss06 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 08:35:29 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:35:29 dc2mds01 kernel: : Lustre: Skipped 27 previous similar messages Mar 5 08:35:29 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394026529 last 1394026529 Mar 5 08:35:29 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 27 previous similar messages Mar 5 08:35:59 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 08:35:59 dc2oss08 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 08:36:23 dc2oss05 kernel: : Lustre: dc2-OST0033: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:36:23 dc2oss05 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 08:36:15 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 08:36:15 dc2oss16 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 08:36:18 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:36:18 dc2oss12 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:36:41 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:36:41 dc2oss10 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 08:37:04 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 08:37:04 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:38:05 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0x67:0x0]] [valid 68719476736] Mar 5 08:38:58 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 08:38:58 dc2oss10 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:39:07 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394026747 last 1394026747 Mar 5 08:39:07 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 133 previous similar messages Mar 5 08:39:19 dc2oss08 kernel: : Lustre: dc2-OST0052: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:39:19 dc2oss08 kernel: : Lustre: Skipped 156 previous similar messages Mar 5 08:39:19 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1394026759 last 1394026759 Mar 5 08:39:19 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 154 previous similar messages Mar 5 08:39:40 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394026780 last 1394026780 Mar 5 08:39:40 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 08:40:00 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394026800 last 1394026800 Mar 5 08:40:00 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 08:40:01 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394026801 last 1394026801 Mar 5 08:40:01 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 08:40:50 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007d_UUID' is not available for connect (no target) Mar 5 08:40:50 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:40:51 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0098: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881807acd800 cur 1394026851 last 1394026851 Mar 5 08:40:51 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 08:40:43 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232051155830, new: 232051155829 replay: 0. see LU-617. Mar 5 08:40:52 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232051172991, new: 232051172990 replay: 0. see LU-617. Mar 5 08:40:56 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394026856 last 1394026856 Mar 5 08:40:56 dc2oss14 kernel: : Lustre: 15632:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 08:40:55 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0019_UUID' is not available for connect (no target) Mar 5 08:40:55 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:41:26 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 08:41:26 dc2oss03 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:41:33 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:41:33 dc2oss03 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 08:41:33 dc2oss03 kernel: : Lustre: 16938:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394026893 last 1394026893 Mar 5 08:41:33 dc2oss03 kernel: : Lustre: 16938:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 08:41:09 dc2oss06 kernel: : Lustre: 15955:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394026869 last 1394026869 Mar 5 08:41:09 dc2oss06 kernel: : Lustre: 15955:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 08:41:19 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394026879 last 1394026879 Mar 5 08:41:19 dc2oss16 kernel: : Lustre: 17170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 08:41:20 dc2oss12 kernel: : Lustre: 9274:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-39), not sending early reply Mar 5 08:41:25 dc2oss12 kernel: : LustreError: 32676:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 644+0s req@ffff88009958d000 x1461679862885272/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 3 to 0 dl 1394026885 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 08:41:25 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:41:25 dc2oss12 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 08:41:25 dc2oss12 kernel: : Lustre: Service thread pid 32676 completed after 644.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 08:41:35 dc2oss10 kernel: : Lustre: 8692:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-55), not sending early reply Mar 5 08:41:35 dc2oss10 kernel: : Lustre: 8692:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Mar 5 08:41:40 dc2oss10 kernel: : LustreError: 15909:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 660+0s req@ffff880a8c44bc00 x1461679862885232/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 3 to 0 dl 1394026900 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 08:41:40 dc2oss10 kernel: : LustreError: 15909:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 08:41:40 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 08:41:40 dc2oss10 kernel: : Lustre: Service thread pid 15909 completed after 660.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 08:41:40 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:41:37 dc2oss05 kernel: : Lustre: 16122:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394026897 last 1394026897 Mar 5 08:41:37 dc2oss05 kernel: : Lustre: 16122:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 08:41:34 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:41:34 dc2oss04 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 08:41:40 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394026900 last 1394026900 Mar 5 08:41:40 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 08:41:53 dc2oss10 kernel: : Lustre: 16369:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394026913 last 1394026913 Mar 5 08:41:53 dc2oss10 kernel: : Lustre: 16369:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 08:42:13 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:42:13 dc2oss02 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 08:42:13 dc2oss02 kernel: : Lustre: 16336:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394026933 last 1394026933 Mar 5 08:42:13 dc2oss02 kernel: : Lustre: 16336:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 08:42:18 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 08:42:18 dc2oss05 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 08:42:17 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:42:17 dc2oss16 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 08:42:17 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 08:42:17 dc2oss12 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 08:42:18 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 08:42:18 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 08:42:46 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 08:42:46 dc2oss13 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 08:42:50 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:42:50 dc2oss11 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 08:43:07 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002c_UUID' is not available for connect (no target) Mar 5 08:43:07 dc2oss06 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 08:43:35 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a2_UUID' is not available for connect (no target) Mar 5 08:43:35 dc2oss15 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 08:43:30 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:43:30 dc2oss01 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 08:43:30 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394027010 last 1394027010 Mar 5 08:43:30 dc2oss01 kernel: : Lustre: 15333:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 08:43:50 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:43:50 dc2oss07 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 08:43:50 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394027030 last 1394027030 Mar 5 08:43:50 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:43:54 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:43:54 dc2oss14 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 08:44:12 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 08:44:12 dc2oss07 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 08:44:19 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:44:19 dc2oss09 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 08:44:20 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 08:44:20 dc2oss09 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 08:44:20 dc2oss13 kernel: : Lustre: dc2-OST007e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:44:20 dc2oss13 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 08:44:22 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 08:44:22 dc2oss01 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 08:45:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 08:45:08 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 08:45:03 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:45:03 dc2oss15 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 08:45:20 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:45:20 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 08:45:33 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 08:45:33 dc2mds01 kernel: : Lustre: Skipped 17 previous similar messages Mar 5 08:45:33 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394027133 last 1394027133 Mar 5 08:45:33 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 17 previous similar messages Mar 5 08:46:14 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 08:46:14 dc2oss08 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 08:46:24 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:46:24 dc2oss05 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 08:46:03 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x116:0x0]] [valid 68719476736] Mar 5 08:46:03 dc2mds01 kernel: : Lustre: 11156:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 08:46:30 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:46:30 dc2oss12 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 08:46:41 dc2oss10 kernel: : Lustre: dc2-OST0062: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:46:41 dc2oss10 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 08:46:43 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 08:46:43 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:47:18 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 08:47:18 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:48:25 dc2mds01 kernel: : Lustre: 1369:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x11b:0x0]] [valid 68719476736] Mar 5 08:49:13 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394027353 last 1394027353 Mar 5 08:49:13 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 08:49:24 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:49:24 dc2oss08 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 08:49:24 dc2oss08 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394027364 last 1394027364 Mar 5 08:49:24 dc2oss08 kernel: : Lustre: 15979:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 08:49:41 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394027381 last 1394027381 Mar 5 08:49:41 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 08:49:50 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005d_UUID' is not available for connect (no target) Mar 5 08:49:50 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 08:50:04 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1394027404 last 1394027404 Mar 5 08:50:04 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 08:50:19 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394027419 last 1394027419 Mar 5 08:50:19 dc2oss04 kernel: : Lustre: 16748:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 08:50:51 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394027451 last 1394027451 Mar 5 08:50:51 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 08:50:58 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394027458 last 1394027458 Mar 5 08:50:58 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 08:50:59 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 08:50:59 dc2oss11 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 08:50:58 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 08:50:58 dc2oss04 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 08:51:15 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394027475 last 1394027475 Mar 5 08:51:15 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 08:51:37 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:51:37 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 08:51:37 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394027497 last 1394027497 Mar 5 08:51:37 dc2oss03 kernel: : Lustre: 16673:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 08:51:42 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 08:51:42 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 08:51:20 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818079be800 cur 1394027480 last 1394027480 Mar 5 08:51:20 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 08:51:31 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 611040, left: 611040, last alive is 48 secs ago Mar 5 08:51:31 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 611040, left: 611040, last alive is 48 secs ago Mar 5 08:51:40 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394027500 last 1394027500 Mar 5 08:51:40 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 08:51:40 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394027500 last 1394027500 Mar 5 08:51:40 dc2oss11 kernel: : Lustre: 14511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 08:51:39 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:51:39 dc2oss04 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:51:53 dc2oss10 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394027513 last 1394027513 Mar 5 08:51:53 dc2oss10 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 08:52:15 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:52:15 dc2oss02 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 08:52:15 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394027535 last 1394027535 Mar 5 08:52:15 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 08:52:21 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 08:52:21 dc2oss05 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 08:52:19 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:52:19 dc2oss16 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 08:52:22 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232052145230, new: 232052145229 replay: 0. see LU-617. Mar 5 08:52:23 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 08:52:23 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 08:52:33 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 94016, left: 94016, last alive is 27 secs ago Mar 5 08:52:33 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 94016, left: 94016, last alive is 27 secs ago Mar 5 08:52:48 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 08:52:48 dc2oss12 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 08:52:52 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0089_UUID' is not available for connect (no target) Mar 5 08:52:52 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 08:52:54 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:52:54 dc2oss11 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 08:53:15 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 08:53:15 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:53:46 dc2oss11 kernel: : Lustre: Service thread pid 25328 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:53:46 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 08:53:46 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:53:46 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394027626.25328 Mar 5 08:53:51 dc2oss11 kernel: : Lustre: Service thread pid 14660 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:53:51 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:53:51 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394027631.14660 Mar 5 08:53:41 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:53:41 dc2oss01 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 08:53:41 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394027621 last 1394027621 Mar 5 08:53:41 dc2oss01 kernel: : Lustre: 15331:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 08:53:51 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 08:53:51 dc2oss15 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 08:53:55 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:53:55 dc2oss14 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 08:53:56 dc2oss07 kernel: : Lustre: dc2-OST0044: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:53:56 dc2oss07 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 08:53:56 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12508400 cur 1394027636 last 1394027636 Mar 5 08:53:56 dc2oss07 kernel: : Lustre: 15871:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 08:54:07 dc2oss11 kernel: : Lustre: Service thread pid 14703 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:54:07 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:54:07 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394027647.14703 Mar 5 08:54:18 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 08:54:18 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 08:54:24 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:54:24 dc2oss13 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 08:54:27 dc2oss09 kernel: : Lustre: dc2-OST0056: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:54:27 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 08:54:57 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 08:54:57 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 08:54:59 dc2oss11 kernel: : Lustre: Service thread pid 25388 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:54:59 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:54:59 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394027699.25388 Mar 5 08:55:12 dc2oss15 kernel: : Lustre: dc2-OST0098: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:55:12 dc2oss15 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 08:55:23 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 08:55:23 dc2oss02 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 08:55:19 dc2oss11 kernel: : Lustre: Service thread pid 14655 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 08:55:19 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 08:55:19 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394027719.14655 Mar 5 08:55:34 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:55:34 dc2oss06 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 08:55:38 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 08:55:38 dc2mds01 kernel: : Lustre: Skipped 30 previous similar messages Mar 5 08:55:38 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394027738 last 1394027738 Mar 5 08:55:38 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 30 previous similar messages Mar 5 08:55:40 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 08:55:40 dc2oss01 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 08:56:08 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 824672, left: 824672, last alive is 0 secs ago Mar 5 08:56:08 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 824672, left: 824672, last alive is 0 secs ago Mar 5 08:56:24 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:56:24 dc2oss05 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 08:56:34 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 618592, left: 618592, last alive is 24 secs ago Mar 5 08:56:34 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 618592, left: 618592, last alive is 24 secs ago Mar 5 08:56:36 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:56:36 dc2oss12 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 08:56:40 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 08:56:40 dc2oss08 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 08:56:45 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:56:45 dc2oss10 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 08:57:02 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 08:57:02 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 08:57:25 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 08:57:25 dc2oss14 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 08:59:20 dc2oss09 kernel: : Lustre: 14507:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394027960 last 1394027960 Mar 5 08:59:20 dc2oss09 kernel: : Lustre: 14507:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 08:59:38 dc2oss08 kernel: : Lustre: dc2-OST004d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 08:59:38 dc2oss08 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 08:59:38 dc2oss08 kernel: : Lustre: 16035:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1394027978 last 1394027978 Mar 5 08:59:38 dc2oss08 kernel: : Lustre: 16035:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 08:59:41 dc2oss12 kernel: : Lustre: 15389:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394027981 last 1394027981 Mar 5 08:59:41 dc2oss12 kernel: : Lustre: 15389:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 09:00:06 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394028006 last 1394028006 Mar 5 09:00:06 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 09:00:20 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 09:00:20 dc2oss10 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:00:25 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394028025 last 1394028025 Mar 5 09:00:25 dc2oss04 kernel: : Lustre: 16646:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 09:00:52 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0099: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf5c5dc00 cur 1394028052 last 1394028052 Mar 5 09:00:52 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 09:01:00 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394028060 last 1394028060 Mar 5 09:01:00 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 09:01:15 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 09:01:15 dc2oss11 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:01:38 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:01:38 dc2oss03 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 09:01:38 dc2oss03 kernel: : Lustre: 17105:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394028098 last 1394028098 Mar 5 09:01:38 dc2oss03 kernel: : Lustre: 17105:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 09:01:31 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394028091 last 1394028091 Mar 5 09:01:31 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 09:01:29 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394028089 last 1394028089 Mar 5 09:01:29 dc2oss16 kernel: : Lustre: 17338:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 09:01:23 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 09:01:23 dc2oss04 kernel: : LustreError: Skipped 41 previous similar messages Mar 5 09:01:42 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394028102 last 1394028102 Mar 5 09:01:42 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 09:01:41 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:01:41 dc2oss04 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 09:01:54 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394028114 last 1394028114 Mar 5 09:01:54 dc2oss11 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 09:01:57 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394028117 last 1394028117 Mar 5 09:01:57 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 09:02:16 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:02:16 dc2oss02 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 09:02:16 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394028136 last 1394028136 Mar 5 09:02:16 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 09:02:29 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:02:29 dc2oss16 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 09:02:35 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 09:02:35 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 09:02:59 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 09:02:59 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:02:56 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:02:56 dc2oss11 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 09:03:16 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002a_UUID' is not available for connect (no target) Mar 5 09:03:16 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 09:03:23 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 09:03:23 dc2oss12 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:03:36 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 09:03:36 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 09:03:42 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:03:42 dc2oss01 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 09:03:42 dc2oss01 kernel: : Lustre: 15308:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394028222 last 1394028222 Mar 5 09:03:42 dc2oss01 kernel: : Lustre: 15308:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 09:03:58 dc2oss14 kernel: : Lustre: dc2-OST008a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:03:58 dc2oss14 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 09:03:59 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 09:03:59 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:04:08 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:04:08 dc2oss07 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:04:08 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1394028248 last 1394028248 Mar 5 09:04:08 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 09:04:14 dc2oss11 kernel: : Lustre: Service thread pid 8755 was inactive for 538.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 09:04:14 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 09:04:14 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394028254.8755 Mar 5 09:04:24 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:04:24 dc2oss13 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 09:04:28 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:04:28 dc2oss09 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 09:04:42 dc2oss11 kernel: : Lustre: Service thread pid 25355 was inactive for 538.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 09:04:42 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 09:04:42 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394028282.25355 Mar 5 09:04:58 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 797792, left: 797792, last alive is 45 secs ago Mar 5 09:04:58 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 797792, left: 797792, last alive is 45 secs ago Mar 5 09:04:48 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:04:48 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 09:05:12 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:05:12 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 09:05:22 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 09:05:22 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:05:35 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 09:05:35 dc2oss07 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:05:37 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:05:37 dc2oss06 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 09:05:38 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 09:05:38 dc2mds01 kernel: : Lustre: Skipped 14 previous similar messages Mar 5 09:05:38 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394028338 last 1394028338 Mar 5 09:05:38 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 14 previous similar messages Mar 5 09:05:46 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 09:05:46 dc2oss01 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:06:07 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 09:06:07 dc2oss02 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:06:34 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:06:34 dc2oss05 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 09:06:38 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:06:38 dc2oss12 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 09:06:46 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 09:06:46 dc2oss08 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 09:06:53 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:06:53 dc2oss10 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 09:07:21 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 09:07:21 dc2oss16 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 09:07:33 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 09:07:33 dc2oss14 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 09:07:37 dc2oss11 kernel: : Lustre: 25426:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-431), not sending early reply Mar 5 09:07:37 dc2oss11 kernel: : Lustre: 25426:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Mar 5 09:07:39 dc2oss11 kernel: : Lustre: 25452:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-412), not sending early reply Mar 5 09:07:42 dc2oss11 kernel: : LustreError: 25328:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1036+0s req@ffff8817fa980050 x1442912236831790/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 4 to 0 dl 1394028462 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:07:42 dc2oss11 kernel: : LustreError: 25328:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 09:07:42 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:07:42 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 09:07:42 dc2oss11 kernel: : Lustre: Service thread pid 25328 completed after 1036.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:07:42 dc2oss11 kernel: : Lustre: 25352:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-431), not sending early reply Mar 5 09:07:44 dc2oss11 kernel: : LustreError: 14703:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1017+0s req@ffff88051e77dc00 x1442912236831802/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394028464 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:07:44 dc2oss11 kernel: : Lustre: Service thread pid 14703 completed after 1017.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:07:47 dc2oss11 kernel: : LustreError: 14660:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1036+0s req@ffff880bd3fde400 x1442912236831793/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 4 to 0 dl 1394028467 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:07:47 dc2oss11 kernel: : Lustre: Service thread pid 14660 completed after 1036.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:08:31 dc2oss11 kernel: : Lustre: 14648:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-412), not sending early reply Mar 5 09:08:36 dc2oss11 kernel: : LustreError: 25388:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1017+0s req@ffff8806427c2c00 x1442912236832491/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394028516 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:08:36 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:08:36 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 09:08:36 dc2oss11 kernel: : Lustre: Service thread pid 25388 completed after 1017.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:08:51 dc2oss11 kernel: : Lustre: 25387:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-412), not sending early reply Mar 5 09:08:56 dc2oss11 kernel: : LustreError: 14655:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 1017+0s req@ffff880bd73fa800 x1442912236832722/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394028536 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:08:56 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:08:56 dc2oss11 kernel: : Lustre: Service thread pid 14655 completed after 1017.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:09:18 dc2mds01 kernel: : Lustre: 11178:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f961:0x386a:0x0]] [valid 68719476736] Mar 5 09:09:34 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394028574 last 1394028574 Mar 5 09:09:34 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 09:09:40 dc2oss11 kernel: : Lustre: 8688:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-264), not sending early reply Mar 5 09:09:45 dc2oss11 kernel: : LustreError: 8755:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 869+0s req@ffff880bd680d800 x1442912236834606/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394028585 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:09:45 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:09:45 dc2oss11 kernel: : Lustre: Service thread pid 8755 completed after 869.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:09:45 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394028585 last 1394028585 Mar 5 09:09:45 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 09:09:39 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:09:39 dc2oss08 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 09:09:39 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394028579 last 1394028579 Mar 5 09:09:39 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 09:10:05 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 09:10:05 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8817b0529050 x1442912236834848/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394028613 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:10:05 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 09:10:05 dc2oss11 kernel: : Lustre: Service thread pid 25355 completed after 861.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:10:06 dc2oss11 kernel: : LustreError: 25413:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd71e7c00 x1442912236840497/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394028919 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:10:06 dc2oss13 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394028606 last 1394028606 Mar 5 09:10:06 dc2oss13 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 09:10:26 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394028626 last 1394028626 Mar 5 09:10:26 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 09:10:31 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 09:10:31 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:10:58 dc2oss15 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394028658 last 1394028658 Mar 5 09:10:58 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394028658 last 1394028658 Mar 5 09:10:58 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 09:11:00 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394028660 last 1394028660 Mar 5 09:11:00 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 09:11:45 dc2oss03 kernel: : Lustre: dc2-OST001a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:11:45 dc2oss03 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 09:11:45 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c03192800 cur 1394028705 last 1394028705 Mar 5 09:11:45 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 09:11:33 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394028693 last 1394028693 Mar 5 09:11:33 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 09:11:31 dc2oss06 kernel: : Lustre: 15837:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1394028691 last 1394028691 Mar 5 09:11:31 dc2oss06 kernel: : Lustre: 15837:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 09:11:30 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232053689223, new: 232053689222 replay: 0. see LU-617. Mar 5 09:11:51 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394028711 last 1394028711 Mar 5 09:11:51 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 09:11:48 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:11:48 dc2oss04 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 09:11:51 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 09:11:51 dc2oss04 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 09:11:55 dc2oss11 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394028715 last 1394028715 Mar 5 09:11:55 dc2oss11 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 09:12:00 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc10e800 cur 1394028720 last 1394028720 Mar 5 09:12:00 dc2oss10 kernel: : Lustre: 15802:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 09:12:27 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:12:27 dc2oss02 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 09:12:27 dc2oss02 kernel: : Lustre: 16336:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394028747 last 1394028747 Mar 5 09:12:27 dc2oss02 kernel: : Lustre: 16336:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 09:12:35 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:12:35 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 09:12:52 dc2oss12 kernel: : Lustre: dc2-OST007c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 09:12:53 dc2oss12 kernel: : LustreError: 14598:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880c00662050 x1461680520618965/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394028814 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:12:53 dc2oss12 kernel: : LustreError: 14598:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 09:12:53 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 09:12:53 dc2oss12 kernel: : LustreError: 9333:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b260ad800 x1461680520618953/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394028813 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:12:53 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 09:12:57 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:12:57 dc2oss11 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 09:13:02 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 09:13:02 dc2oss11 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 09:13:02 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 09:13:02 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 09:13:18 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 09:13:18 dc2oss06 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:13:46 149.165.238.1 kernel: kernel: [4303198.544046] Lustre: 23632:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669104396043 sent from dc2-OST0023-osc-ffff880956851000 to NID 10.10.0.4@o2ib 17s ago has timed out (17s prior to deadline). Mar 5 09:13:46 149.165.238.1 kernel: kernel: [4303198.544062] Lustre: dc2-OST0023-osc-ffff880956851000: Connection to service dc2-OST0023 via nid 10.10.0.4@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 09:13:46 149.165.238.1 kernel: kernel: [4303198.544389] Lustre: dc2-OST0023-osc-ffff880956851000: Connection restored to service dc2-OST0023 using nid 10.10.0.4@o2ib. Mar 5 09:13:46 149.165.238.1 kernel: kernel: [4303198.544399] Lustre: Server dc2-OST0023_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 09:13:56 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 09:13:56 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:13:43 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 09:13:43 dc2oss12 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:13:49 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:13:49 dc2oss01 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 09:13:49 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394028829 last 1394028829 Mar 5 09:13:49 dc2oss01 kernel: : Lustre: 15517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 09:14:05 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:14:05 dc2oss14 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 09:14:07 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 09:14:07 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:14:09 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:14:09 dc2oss07 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 09:14:09 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394028849 last 1394028849 Mar 5 09:14:09 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 09:14:12 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 09:14:12 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:14:24 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:14:24 dc2oss13 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:14:28 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:14:28 dc2oss09 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 09:14:48 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:14:48 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 20 previous similar messages Mar 5 09:15:12 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:15:12 dc2oss15 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 09:15:18 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 272320, left: 272320, last alive is 35 secs ago Mar 5 09:15:18 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 272320, left: 272320, last alive is 35 secs ago Mar 5 09:15:47 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 09:15:47 dc2oss07 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:15:48 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:15:48 dc2oss06 kernel: : Lustre: Skipped 142 previous similar messages Mar 5 09:16:08 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 09:16:08 dc2oss09 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 09:16:00 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 09:16:00 dc2oss01 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:16:08 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:16:08 dc2mds01 kernel: : Lustre: Skipped 30 previous similar messages Mar 5 09:16:08 dc2mds01 kernel: : Lustre: 11166:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394028968 last 1394028968 Mar 5 09:16:08 dc2mds01 kernel: : Lustre: 11166:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 30 previous similar messages Mar 5 09:16:39 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 09:16:39 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:16:37 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:16:37 dc2oss05 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 09:16:38 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:16:38 dc2oss12 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 09:16:46 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 431840, left: 431840, last alive is 2 secs ago Mar 5 09:16:46 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 431840, left: 431840, last alive is 2 secs ago Mar 5 09:17:00 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:17:00 dc2oss10 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 09:17:01 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 09:17:01 dc2oss08 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:17:24 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 09:17:24 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:17:40 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 09:17:40 dc2oss14 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 09:19:38 dc2oss09 kernel: : Lustre: 16461:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394029178 last 1394029178 Mar 5 09:19:38 dc2oss09 kernel: : Lustre: 16461:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 09:19:31 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x12d:0x0]] [valid 68719476736] Mar 5 09:19:40 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:19:40 dc2oss08 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 09:19:40 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394029180 last 1394029180 Mar 5 09:19:40 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 09:19:52 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394029192 last 1394029192 Mar 5 09:19:52 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 09:20:11 dc2oss13 kernel: : Lustre: 14428:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394029211 last 1394029211 Mar 5 09:20:11 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394029211 last 1394029211 Mar 5 09:20:11 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 09:20:04 dc2mds01 kernel: : Lustre: 11178:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x135:0x0]] [valid 68719476736] Mar 5 09:20:06 dc2mds01 kernel: : Lustre: 11130:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:20:06 dc2mds01 kernel: : Lustre: 11130:0:(mdd_device.c:900:obf_lookup()) Skipped 270 previous similar messages Mar 5 09:20:11 dc2mds01 kernel: : Lustre: 11130:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:20:11 dc2mds01 kernel: : Lustre: 11130:0:(mdd_device.c:900:obf_lookup()) Skipped 3 previous similar messages Mar 5 09:20:20 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:20:20 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) Skipped 7 previous similar messages Mar 5 09:20:30 dc2oss04 kernel: : Lustre: 16649:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394029230 last 1394029230 Mar 5 09:20:30 dc2oss04 kernel: : Lustre: 16649:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 09:20:37 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:20:37 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) Skipped 10 previous similar messages Mar 5 09:20:43 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 09:20:43 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:20:45 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 726112, left: 726112, last alive is 36 secs ago Mar 5 09:20:45 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 726112, left: 726112, last alive is 36 secs ago Mar 5 09:20:58 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394029258 last 1394029258 Mar 5 09:20:58 dc2oss15 kernel: : Lustre: 15772:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 09:21:00 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394029260 last 1394029260 Mar 5 09:21:00 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 09:20:55 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x13f:0x0]] [valid 68719476736] Mar 5 09:20:55 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 09:21:47 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:21:47 dc2oss03 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 09:21:47 dc2oss03 kernel: : Lustre: 17005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1394029307 last 1394029307 Mar 5 09:21:47 dc2oss03 kernel: : Lustre: 17005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 09:21:23 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:21:23 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) Skipped 22 previous similar messages Mar 5 09:21:37 dc2oss16 kernel: : Lustre: 17124:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394029297 last 1394029297 Mar 5 09:21:37 dc2oss16 kernel: : Lustre: 17124:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 09:21:47 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394029307 last 1394029307 Mar 5 09:21:47 dc2oss06 kernel: : Lustre: 15893:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 09:22:02 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394029322 last 1394029322 Mar 5 09:22:02 dc2oss10 kernel: : Lustre: 15820:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 09:22:05 dc2oss04 kernel: : Lustre: dc2-OST0029: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:22:05 dc2oss04 kernel: : Lustre: dc2-OST0022: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:22:05 dc2oss04 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 09:22:05 dc2oss05 kernel: : Lustre: 15813:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394029325 last 1394029325 Mar 5 09:22:05 dc2oss05 kernel: : Lustre: 15813:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 09:22:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394029335 last 1394029335 Mar 5 09:22:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 09:22:13 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0017_UUID' is not available for connect (no target) Mar 5 09:22:13 dc2oss04 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 09:22:28 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:22:28 dc2oss02 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 09:22:28 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394029348 last 1394029348 Mar 5 09:22:28 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 09:22:27 dc2mds01 kernel: : Lustre: 5943:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:22:27 dc2mds01 kernel: : Lustre: 5943:0:(mdd_device.c:900:obf_lookup()) Skipped 38 previous similar messages Mar 5 09:22:31 dc2mds01 kernel: : Lustre: 11173:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x14c:0x0]] [valid 68719476736] Mar 5 09:22:31 dc2mds01 kernel: : Lustre: 11173:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 09:22:39 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:22:39 dc2oss16 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 09:23:04 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 708192, left: 708192, last alive is 25 secs ago Mar 5 09:23:04 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 708192, left: 708192, last alive is 25 secs ago Mar 5 09:23:01 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:23:01 dc2oss11 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 09:23:05 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 09:23:05 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:23:04 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 09:23:04 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:23:15 dc2oss11 kernel: : Lustre: Service thread pid 25438 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 09:23:15 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 09:23:15 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394029395.25438 Mar 5 09:23:26 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 09:23:26 dc2oss06 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 09:24:12 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 09:24:12 dc2oss03 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 09:24:00 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 09:24:00 dc2oss12 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:24:00 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:24:00 dc2oss01 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 09:24:00 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394029440 last 1394029440 Mar 5 09:24:00 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 09:24:09 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:24:09 dc2oss07 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 09:24:09 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394029449 last 1394029449 Mar 5 09:24:09 dc2oss07 kernel: : Lustre: 14323:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 09:24:12 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 09:24:12 dc2oss15 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 09:24:15 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:24:15 dc2oss14 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 09:24:28 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:24:28 dc2oss13 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 09:24:39 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:24:39 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:24:38 dc2mds01 kernel: : Lustre: 14548:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:24:38 dc2mds01 kernel: : Lustre: 14548:0:(mdd_device.c:900:obf_lookup()) Skipped 85 previous similar messages Mar 5 09:24:45 dc2mds01 kernel: : LustreError: 6341:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232054743146, new: 232054743145 replay: 0. see LU-617. Mar 5 09:25:11 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 09:25:11 dc2oss13 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 09:25:17 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:25:17 dc2oss15 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 09:25:33 dc2oss11 kernel: : Lustre: Service thread pid 25374 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 09:25:33 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 09:25:33 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394029533.25374 Mar 5 09:26:01 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:26:01 dc2oss06 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 09:26:03 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:26:03 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 09:26:08 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 09:26:08 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 09:26:15 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 09:26:15 dc2mds01 kernel: : Lustre: Skipped 25 previous similar messages Mar 5 09:26:15 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394029575 last 1394029575 Mar 5 09:26:15 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 25 previous similar messages Mar 5 09:26:23 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x155:0x0]] [valid 68719476736] Mar 5 09:26:23 dc2mds01 kernel: : Lustre: 5943:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 1 previous similar message Mar 5 09:26:27 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 09:26:27 dc2oss07 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 09:26:40 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 09:26:40 dc2oss02 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 09:26:34 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0063_UUID' is not available for connect (no target) Mar 5 09:26:34 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 09:26:41 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:26:41 dc2oss12 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 09:26:40 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:26:40 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:26:40 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:26:40 dc2oss05 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:26:50 dc2oss11 kernel: : Lustre: 25426:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 09:26:53 dc2oss11 kernel: : Lustre: 14704:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 09:26:55 dc2oss11 kernel: : LustreError: 8722:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff88089c69b800 x1442912236846699/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394029615 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:26:55 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:26:55 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 09:26:58 dc2oss11 kernel: : LustreError: 8735:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd725b400 x1442912236846704/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394029618 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:26:59 dc2oss11 kernel: : Lustre: 8722:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 09:27:01 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 09:27:01 dc2oss11 kernel: : LustreError: 25454:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd68a0c00 x1442912236847665/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394029706 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:27:01 dc2oss11 kernel: : LustreError: 25374:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8802ee82f000 x1442912236851886/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394029640 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:27:01 dc2oss11 kernel: : LustreError: 25374:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 09:27:01 dc2oss11 kernel: : Lustre: Service thread pid 25374 completed after 288.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 09:27:05 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:27:05 dc2oss10 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 09:27:11 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 09:27:11 dc2oss08 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:27:17 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 708192, left: 708192, last alive is 46 secs ago Mar 5 09:27:17 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 708192, left: 708192, last alive is 46 secs ago Mar 5 09:27:37 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 09:27:37 dc2oss16 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 09:28:06 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 09:28:06 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:28:50 dc2oss11 kernel: : LustreError: 8808:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 100+0s req@ffff880bd6f68400 x1442912236854847/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394029730 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 09:28:50 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:28:50 dc2oss11 kernel: : Lustre: Skipped 7 previous similar messages Mar 5 09:29:36 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 369824, left: 369824, last alive is 0 secs ago Mar 5 09:29:36 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 369824, left: 369824, last alive is 0 secs ago Mar 5 09:29:39 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394029779 last 1394029779 Mar 5 09:29:39 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 09:29:51 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from c5cd8aef-e176-83a7-f100-808af2d0f0ad@149.165.228.223@tcp t0 exp (null) cur 1394029791 last 0 Mar 5 09:29:51 dc2oss08 kernel: : Lustre: 14449:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from c5cd8aef-e176-83a7-f100-808af2d0f0ad@149.165.228.223@tcp t0 exp (null) cur 1394029791 last 0 Mar 5 09:29:51 dc2oss08 kernel: : Lustre: 14449:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 09:29:56 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394029796 last 1394029796 Mar 5 09:29:56 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 09:29:57 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:29:57 dc2mds01 kernel: : Lustre: 1386:0:(mdd_device.c:900:obf_lookup()) Skipped 58 previous similar messages Mar 5 09:30:13 dc2oss13 kernel: : Lustre: 15836:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394029813 last 1394029813 Mar 5 09:30:13 dc2oss13 kernel: : Lustre: 15836:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 09:30:14 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 627904, left: 627904, last alive is 24 secs ago Mar 5 09:30:14 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 627904, left: 627904, last alive is 24 secs ago Mar 5 09:30:27 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:30:27 dc2oss08 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 09:30:39 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394029839 last 1394029839 Mar 5 09:30:39 dc2oss04 kernel: : Lustre: 16743:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 09:30:44 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 09:30:44 dc2oss10 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 09:31:00 dc2oss15 kernel: : Lustre: 15755:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394029860 last 1394029860 Mar 5 09:31:00 dc2oss15 kernel: : Lustre: 15755:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 09:31:05 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394029865 last 1394029865 Mar 5 09:31:05 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 09:31:11 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 7 active RPCs Mar 5 09:31:11 dc2oss11 kernel: : LustreError: 25474:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc71a6400 x1442912236856069/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394030517 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:31:11 dc2oss11 kernel: : LustreError: 25474:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 09:31:11 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:31:13 dc2xfer02 kernel: : Lustre: 2702:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:31:13 dc2xfer02 kernel: : Lustre: 2702:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:31:17 dc2xfer01 kernel: : Lustre: 2780:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:31:17 dc2xfer01 kernel: : Lustre: 2780:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:31:48 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:31:48 dc2oss03 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 09:31:48 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394029908 last 1394029908 Mar 5 09:31:48 dc2oss03 kernel: : Lustre: 16996:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 09:31:27 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f961:0x3888:0x0]] [valid 68719476736] Mar 5 09:31:27 dc2mds01 kernel: : Lustre: 1386:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 7 previous similar messages Mar 5 09:31:40 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394029900 last 1394029900 Mar 5 09:31:40 dc2oss16 kernel: : Lustre: 17240:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 09:31:51 dc2oss06 kernel: : Lustre: 16029:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394029911 last 1394029911 Mar 5 09:31:51 dc2oss06 kernel: : Lustre: 16029:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 09:32:07 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394029927 last 1394029927 Mar 5 09:32:07 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 09:32:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394029935 last 1394029935 Mar 5 09:32:15 dc2oss11 kernel: : Lustre: 14505:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 09:32:30 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:32:30 dc2oss02 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 09:32:30 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394029950 last 1394029950 Mar 5 09:32:30 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 09:32:25 dc2oss04 kernel: : Lustre: dc2-OST0029: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:32:25 dc2oss04 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 09:32:30 dc2oss05 kernel: : Lustre: 15901:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394029950 last 1394029950 Mar 5 09:32:30 dc2oss05 kernel: : Lustre: 15901:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 134 previous similar messages Mar 5 09:32:36 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0016_UUID' is not available for connect (no target) Mar 5 09:32:36 dc2oss04 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 09:32:43 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:32:43 dc2oss16 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 09:33:11 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 46528, left: 46528, last alive is 37 secs ago Mar 5 09:33:11 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 46528, left: 46528, last alive is 37 secs ago Mar 5 09:33:17 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 09:33:17 dc2oss11 kernel: : LustreError: Skipped 43 previous similar messages Mar 5 09:33:24 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:33:24 dc2oss11 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 09:33:32 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 09:33:32 dc2oss06 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 09:34:18 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 09:34:18 dc2oss03 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:34:06 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003e_UUID' is not available for connect (no target) Mar 5 09:34:06 dc2oss05 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:34:01 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:34:01 dc2oss01 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 09:34:01 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394030041 last 1394030041 Mar 5 09:34:01 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 09:34:12 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:34:12 dc2oss07 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:34:12 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394030052 last 1394030052 Mar 5 09:34:12 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 09:34:17 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 09:34:17 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:34:23 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:34:23 dc2oss14 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 09:34:23 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 09:34:23 dc2oss12 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 09:34:31 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:34:31 dc2oss13 kernel: : Lustre: Skipped 135 previous similar messages Mar 5 09:34:39 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:34:39 dc2oss09 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 09:35:20 dc2oss15 kernel: : Lustre: dc2-OST0099: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:35:20 dc2oss15 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 09:35:42 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0092_UUID' is not available for connect (no target) Mar 5 09:35:42 dc2oss13 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 09:35:55 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 45152, left: 45152, last alive is 2 secs ago Mar 5 09:35:55 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 45152, left: 45152, last alive is 2 secs ago Mar 5 09:36:09 dc2oss02 kernel: : Lustre: dc2-OST0013: Client 70ceee40-24a5-19e7-a31d-aae72e3e3e58 (at 149.165.229.30@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 09:36:09 dc2oss02 kernel: : LustreError: 21227:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf79d8850 x1461681120992884/t0(0) o3->70ceee40-24a5-19e7-a31d-aae72e3e3e58@149.165.229.30@tcp:0/0 lens 464/400 e 1 to 0 dl 1394030209 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:36:09 dc2oss02 kernel: : LustreError: 21227:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 09:36:12 dc2oss02 kernel: : Lustre: dc2-OST0013: Bulk IO read error with 70ceee40-24a5-19e7-a31d-aae72e3e3e58 (at 149.165.229.30@tcp), client will retry: rc -110 Mar 5 09:36:12 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 09:36:08 dc2oss06 kernel: : Lustre: dc2-OST003d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:36:08 dc2oss06 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 09:36:20 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 556928, left: 556928, last alive is 2 secs ago Mar 5 09:36:20 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 556928, left: 556928, last alive is 2 secs ago Mar 5 09:36:28 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 09:36:28 dc2oss07 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 09:36:36 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 09:36:36 dc2oss09 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:36:54 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 09:36:54 dc2oss02 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:36:40 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:36:40 dc2oss05 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 09:36:49 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:36:49 dc2oss12 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 09:36:41 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:36:41 dc2mds01 kernel: : Lustre: Skipped 23 previous similar messages Mar 5 09:36:41 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394030201 last 1394030201 Mar 5 09:36:41 dc2mds01 kernel: : Lustre: 11156:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 25 previous similar messages Mar 5 09:36:47 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 09:36:47 dc2oss01 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 09:37:22 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:37:22 dc2oss10 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 09:37:22 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0047_UUID' is not available for connect (no target) Mar 5 09:37:22 dc2oss08 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 09:37:45 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:37:45 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 09:38:17 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0098_UUID' is not available for connect (no target) Mar 5 09:38:17 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 09:38:19 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 09:38:19 dc2oss14 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 09:38:35 dc2mds01 kernel: : Lustre: 11173:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021be206:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 09:38:35 dc2mds01 kernel: : Lustre: 11173:0:(mdd_device.c:900:obf_lookup()) Skipped 235 previous similar messages Mar 5 09:39:40 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c00a7e400 cur 1394030380 last 1394030380 Mar 5 09:39:40 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 09:39:53 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1394030393 last 1394030393 Mar 5 09:39:53 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 09:39:56 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c962800 cur 1394030396 last 1394030396 Mar 5 09:39:56 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 09:40:15 dc2oss13 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394030415 last 1394030415 Mar 5 09:40:15 dc2oss13 kernel: : Lustre: 16125:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 09:40:33 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:40:33 dc2oss08 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 09:40:44 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005b_UUID' is not available for connect (no target) Mar 5 09:40:44 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 09:40:47 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394030447 last 1394030447 Mar 5 09:40:47 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 09:41:06 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394030466 last 1394030466 Mar 5 09:41:06 dc2oss14 kernel: : Lustre: 14550:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 09:41:08 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394030468 last 1394030468 Mar 5 09:41:08 dc2oss15 kernel: : Lustre: 14568:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 09:41:42 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394030502 last 1394030502 Mar 5 09:41:42 dc2oss16 kernel: : Lustre: 17358:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 09:42:07 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:42:07 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 09:42:07 dc2oss03 kernel: : Lustre: 17051:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394030527 last 1394030527 Mar 5 09:42:07 dc2oss03 kernel: : Lustre: 17051:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 09:42:04 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394030524 last 1394030524 Mar 5 09:42:04 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 09:42:09 dc2oss10 kernel: : Lustre: 16506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc181400 cur 1394030529 last 1394030529 Mar 5 09:42:09 dc2oss10 kernel: : Lustre: 16506:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 09:42:17 dc2oss11 kernel: : Lustre: 15532:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394030537 last 1394030537 Mar 5 09:42:17 dc2oss11 kernel: : Lustre: 15532:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 09:42:39 dc2oss02 kernel: : Lustre: dc2-OST000e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:42:39 dc2oss02 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 09:42:39 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1394030559 last 1394030559 Mar 5 09:42:39 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 09:42:26 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:42:26 dc2oss04 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 09:42:34 dc2oss05 kernel: : Lustre: 15901:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394030554 last 1394030554 Mar 5 09:42:34 dc2oss05 kernel: : Lustre: 15901:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 135 previous similar messages Mar 5 09:42:44 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001a_UUID' is not available for connect (no target) Mar 5 09:42:44 dc2oss04 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 09:42:51 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:42:51 dc2oss16 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 09:43:20 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007d_UUID' is not available for connect (no target) Mar 5 09:43:20 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 09:43:37 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 09:43:37 dc2oss06 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 09:43:43 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:43:43 dc2oss11 kernel: : Lustre: Skipped 133 previous similar messages Mar 5 09:44:14 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:44:14 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 09:44:14 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394030654 last 1394030654 Mar 5 09:44:14 dc2oss07 kernel: : Lustre: 14344:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 09:44:12 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:44:12 dc2oss01 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 09:44:12 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394030652 last 1394030652 Mar 5 09:44:12 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 09:44:24 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:44:24 dc2oss14 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 09:44:33 dc2oss11 kernel: : Lustre: 25320:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 09:44:29 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 09:44:29 dc2oss15 kernel: : LustreError: Skipped 40 previous similar messages Mar 5 09:44:35 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:44:35 dc2oss13 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 09:44:38 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006e_UUID' is not available for connect (no target) Mar 5 09:44:38 dc2oss12 kernel: : LustreError: Skipped 40 previous similar messages Mar 5 09:44:38 dc2oss11 kernel: : LustreError: 25335:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd648b800 x1442912236857716/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394030678 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:44:38 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:44:38 dc2oss11 kernel: : Lustre: Skipped 6 previous similar messages Mar 5 09:44:42 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 8 active RPCs Mar 5 09:44:42 dc2oss11 kernel: : LustreError: 14647:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd452a000 x1442912236865334/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394031436 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:44:42 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 09:44:42 dc2oss11 kernel: : LustreError: 14647:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 7 previous similar messages Mar 5 09:44:42 dc2oss11 kernel: : LustreError: 25433:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd377b400 x1442912236859620/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394030868 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:44:42 dc2oss11 kernel: : LustreError: 25433:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 09:44:43 dc2oss11 kernel: : LustreError: 8631:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff88113c1e9050 x1442912236858180/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394030715 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 09:44:43 dc2oss11 kernel: : LustreError: 8631:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 09:44:45 dc2xfer01 kernel: : Lustre: 2787:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:44:45 dc2xfer01 kernel: : Lustre: 2787:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:44:47 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:44:47 dc2oss09 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 09:44:50 dc2xfer02 kernel: : Lustre: 2702:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:44:50 dc2xfer02 kernel: : Lustre: 2702:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 09:44:51 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 09:44:51 dc2oss05 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 09:45:29 dc2oss15 kernel: : Lustre: dc2-OST0098: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:45:29 dc2oss15 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 09:45:57 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 09:45:57 dc2oss03 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:46:10 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:46:10 dc2oss06 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 09:46:14 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 09:46:14 dc2oss13 kernel: : LustreError: Skipped 41 previous similar messages Mar 5 09:46:37 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST005f_UUID' is not available for connect (no target) Mar 5 09:46:37 dc2oss09 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 09:46:26 dc2mds01 kernel: : Lustre: 11173:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x19f:0x0]] [valid 68719476736] Mar 5 09:46:26 dc2mds01 kernel: : Lustre: 11173:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages Mar 5 09:46:47 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 09:46:47 dc2oss07 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:46:48 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:46:48 dc2oss05 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 09:47:04 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:47:04 dc2oss12 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 09:46:53 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 09:46:53 dc2mds01 kernel: : Lustre: Skipped 31 previous similar messages Mar 5 09:46:53 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394030813 last 1394030813 Mar 5 09:46:53 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages Mar 5 09:47:26 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 09:47:26 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:47:19 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 09:47:19 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:47:44 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 09:47:44 dc2oss08 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:47:52 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:47:52 dc2oss10 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 09:48:27 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 09:48:27 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 09:48:45 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 09:48:45 dc2oss16 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 09:49:00 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:49:00 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 23 previous similar messages Mar 5 09:49:30 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232056834735, new: 232056834734 replay: 0. see LU-617. Mar 5 09:49:42 dc2oss09 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394030982 last 1394030982 Mar 5 09:49:42 dc2oss09 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 09:49:59 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181bfd4400 cur 1394030999 last 1394030999 Mar 5 09:49:59 dc2oss12 kernel: : Lustre: 15400:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 09:50:18 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394031018 last 1394031018 Mar 5 09:50:18 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 09:50:25 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394031025 last 1394031025 Mar 5 09:50:25 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 09:50:35 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:50:35 dc2oss08 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 09:50:47 dc2oss04 kernel: : Lustre: 16681:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394031047 last 1394031047 Mar 5 09:50:47 dc2oss04 kernel: : Lustre: 16681:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 09:50:57 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 09:50:57 dc2oss10 kernel: : LustreError: Skipped 40 previous similar messages Mar 5 09:51:07 dc2oss14 kernel: : Lustre: 15718:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394031067 last 1394031067 Mar 5 09:51:07 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394031067 last 1394031067 Mar 5 09:51:07 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 09:51:16 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0099: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf5c5dc00 cur 1394031076 last 1394031076 Mar 5 09:51:16 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 09:51:42 dc2oss16 kernel: : Lustre: 17252:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818079be800 cur 1394031102 last 1394031102 Mar 5 09:51:42 dc2oss16 kernel: : Lustre: 17252:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 09:52:08 dc2oss03 kernel: : Lustre: dc2-OST0019: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:52:08 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 09:52:08 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0019: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c038c2400 cur 1394031128 last 1394031128 Mar 5 09:52:08 dc2oss03 kernel: : Lustre: 17126:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 09:52:09 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394031129 last 1394031129 Mar 5 09:52:09 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 09:52:09 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc10e800 cur 1394031129 last 1394031129 Mar 5 09:52:09 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 09:52:18 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394031138 last 1394031138 Mar 5 09:52:18 dc2oss11 kernel: : Lustre: 15503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 09:52:40 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:52:40 dc2oss02 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 09:52:40 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394031160 last 1394031160 Mar 5 09:52:40 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 09:52:40 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394031160 last 1394031160 Mar 5 09:52:40 dc2oss05 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 09:52:37 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:52:37 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:52:37 dc2oss04 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 09:52:37 dc2oss04 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 09:52:49 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 09:52:49 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 09:52:59 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:52:59 dc2oss16 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 09:53:45 dc2oss11 kernel: : Lustre: dc2-OST0072: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:53:45 dc2oss11 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 09:53:52 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 09:53:52 dc2oss11 kernel: : LustreError: Skipped 10 previous similar messages Mar 5 09:53:51 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002d_UUID' is not available for connect (no target) Mar 5 09:53:51 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:54:19 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:54:19 dc2oss07 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 09:54:19 dc2oss07 kernel: : Lustre: 15814:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394031259 last 1394031258 Mar 5 09:54:19 dc2oss07 kernel: : Lustre: 15814:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 09:54:17 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:54:17 dc2oss01 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 09:54:17 dc2oss01 kernel: : Lustre: 15407:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394031257 last 1394031257 Mar 5 09:54:17 dc2oss01 kernel: : Lustre: 15407:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 09:54:27 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:54:27 dc2oss14 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 09:54:40 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:54:40 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 09:54:40 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 09:54:40 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:54:48 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:54:48 dc2oss09 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 09:54:49 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 09:54:49 dc2oss12 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:55:07 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003b_UUID' is not available for connect (no target) Mar 5 09:55:07 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 09:55:35 dc2oss15 kernel: : Lustre: dc2-OST0095: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:55:35 dc2oss15 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 09:55:59 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0024_UUID' is not available for connect (no target) Mar 5 09:55:59 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 09:56:10 dc2oss06 kernel: : Lustre: dc2-OST003e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:56:10 dc2oss06 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 09:56:26 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 09:56:26 dc2oss13 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 09:56:39 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 09:56:39 dc2oss09 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 09:56:50 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:56:50 dc2oss05 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 09:57:05 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:57:05 dc2oss12 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 09:57:26 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 09:57:26 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 09:57:21 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f957:0x2d6:0x0]] [valid 68719476736] Mar 5 09:57:21 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 10 previous similar messages Mar 5 09:57:56 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 09:57:56 dc2oss10 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 09:58:05 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 09:58:05 dc2oss01 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 09:58:45 dc2oss03 kernel: : Lustre: 16938:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394031518/real 1394031518] req@ffff8806b5442400 x1461670870642394/t0(0) o104->dc2-OST001f@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394031525 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 09:58:45 dc2oss03 kernel: : LustreError: 138-a: dc2-OST001f: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 09:58:39 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 09:58:39 dc2oss14 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 09:59:05 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 09:59:05 dc2oss16 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 09:59:16 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 858944, left: 858944, last alive is 46 secs ago Mar 5 09:59:16 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 858944, left: 858944, last alive is 46 secs ago Mar 5 09:58:59 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 09:58:59 dc2mds01 kernel: : Lustre: Skipped 20 previous similar messages Mar 5 09:58:59 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394031539 last 1394031539 Mar 5 09:58:59 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 20 previous similar messages Mar 5 09:59:22 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0x79:0x0]] [valid 68719476736] Mar 5 09:59:09 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 09:59:09 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 7 previous similar messages Mar 5 09:59:48 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394031588 last 1394031588 Mar 5 09:59:48 dc2oss09 kernel: : Lustre: 16170:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 10:00:10 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 10:00:10 dc2oss07 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 10:00:10 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 10:00:10 dc2oss08 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:00:06 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394031606 last 1394031606 Mar 5 10:00:06 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 10:00:22 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394031622 last 1394031622 Mar 5 10:00:22 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 10:00:24 dc2oss11 kernel: : Lustre: 16209:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394031617/real 0] req@ffff881371c40800 x1456766130989601/t0(0) o104->dc2-OST0071@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394031624 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 10:00:24 dc2oss11 kernel: : Lustre: 16209:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 5 10:00:24 dc2oss11 kernel: : LustreError: 138-a: dc2-OST0071: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 10:00:24 dc2oss11 kernel: : LustreError: 14476:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8809a4680800 x1456766130989620/t0(0) o105->dc2-OST0071@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 Mar 5 10:00:24 dc2oss11 kernel: : LustreError: 14476:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.206@tcp) returned 0 from completion AST ns: filter-dc2-OST0071_UUID lock: ffff880ed1e78480/0x309c239b3adce7b0 lrc: 3/0,0 mode: PW/PW res: 22484368/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x0 remote: 0xfd05479c5773002e expref: 164 pid: 16209 timeout 0 Mar 5 10:00:25 dc2oss11 kernel: : LustreError: 8423:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.226.206@tcp arrived at 1394031625 with bad export cookie 3502713759796586857 Mar 5 10:00:25 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394031625 last 1394031625 Mar 5 10:00:25 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 63 previous similar messages Mar 5 10:00:36 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:00:36 dc2oss08 kernel: : Lustre: Skipped 64 previous similar messages Mar 5 10:00:50 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394031650 last 1394031650 Mar 5 10:00:50 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 10:01:08 dc2oss14 kernel: : Lustre: 15815:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394031668 last 1394031668 Mar 5 10:01:08 dc2oss14 kernel: : Lustre: 15815:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 10:01:21 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394031681 last 1394031681 Mar 5 10:01:21 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394031681 last 1394031681 Mar 5 10:01:21 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 10:01:44 dc2oss11 kernel: : Lustre: Service thread pid 8820 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:01:44 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:01:44 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394031704.8820 Mar 5 10:01:47 dc2oss11 kernel: : Lustre: Service thread pid 14648 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:01:47 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:01:47 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394031707.14648 Mar 5 10:02:08 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:02:08 dc2oss03 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 10:02:08 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394031728 last 1394031728 Mar 5 10:02:08 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 10:01:56 dc2oss11 kernel: : Lustre: Service thread pid 14703 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:01:56 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:01:56 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394031716.14703 Mar 5 10:02:00 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394031720 last 1394031720 Mar 5 10:02:00 dc2oss16 kernel: : Lustre: 17232:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 10:02:16 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 10:02:16 dc2oss10 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:02:14 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394031734 last 1394031733 Mar 5 10:02:14 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 10:02:30 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394031750 last 1394031750 Mar 5 10:02:30 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 10:02:29 dc2oss10 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1394031749 last 1394031749 Mar 5 10:02:29 dc2oss10 kernel: : Lustre: 16523:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 10:02:55 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:02:55 dc2oss02 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 10:02:55 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394031775 last 1394031775 Mar 5 10:02:55 dc2oss02 kernel: : Lustre: 16278:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 10:02:44 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0030: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be2951800 cur 1394031764 last 1394031764 Mar 5 10:02:44 dc2oss05 kernel: : Lustre: 15951:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 10:02:54 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:02:54 dc2oss04 kernel: : Lustre: dc2-OST0029: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:02:54 dc2oss04 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 10:02:54 dc2oss04 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 10:02:55 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 10:02:55 dc2oss04 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:03:03 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:03:03 dc2oss16 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 10:03:25 dc2oss07 kernel: : Lustre: dc2-OST0046: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:03:26 dc2oss07 kernel: : LustreError: 6190:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b837e9000 x1461680445215952/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031839 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:26 dc2oss07 kernel: : Lustre: dc2-OST0046: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:25 dc2oss09 kernel: : Lustre: dc2-OST005e: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 10:03:26 dc2oss09 kernel: : LustreError: 22447:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806bd970400 x1461680445215919/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031835 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:26 dc2oss09 kernel: : LustreError: 22447:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:03:26 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:26 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:26 dc2oss09 kernel: : LustreError: 14661:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881809475050 x1461680445215937/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031837 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:26 dc2oss09 kernel: : LustreError: 14661:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:03:26 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:26 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:03:30 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:03:30 dc2oss10 kernel: : LustreError: 15928:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fe936850 x1461680445215941/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031845 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:30 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:30 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:31 dc2oss10 kernel: : LustreError: 8637:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd79d0800 x1461680445215927/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031844 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:31 dc2oss10 kernel: : LustreError: 8637:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:03:35 dc2oss11 kernel: : Lustre: dc2-OST006c: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 10:03:35 dc2oss11 kernel: : LustreError: 8788:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fae52850 x1461680445215929/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394032084 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:35 dc2oss11 kernel: : LustreError: 8788:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 10:03:35 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:35 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 10:03:36 dc2oss11 kernel: : LustreError: 25363:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd60e4400 x1461680445215930/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394032085 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:03:36 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:03:36 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:03:38 dc2mds01 kernel: : Lustre: 9828:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394031807/real 1394031807] req@ffff882f1a09f000 x1461670450844721/t0(0) o104->dc2-MDT0000@149.165.229.55@tcp:15/16 lens 296/192 e 0 to 1 dl 1394031818 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 10:03:38 dc2mds01 kernel: : Lustre: 9828:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Mar 5 10:03:38 dc2mds01 kernel: : LustreError: 138-a: dc2-MDT0000: A client on nid 149.165.229.55@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 10:03:41 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 1003872, left: 1003872, last alive is 4 secs ago Mar 5 10:03:41 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 1003872, left: 1003872, last alive is 4 secs ago Mar 5 10:04:03 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:04:04 dc2oss03 kernel: : LustreError: 16763:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881805ae3850 x1461679860059109/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031870 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:04:04 dc2oss03 kernel: : LustreError: 16763:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 10:04:04 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 10:04:04 dc2oss03 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 10:03:43 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd_cb.c:2520:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 149.165.229.55@tcp detected; the oldest(ffff880ffef2c000) timed out 1 secs ago, resid: 370304, wmem: 652800 Mar 5 10:03:43 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd_cb.c:2520:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer 149.165.229.55@tcp detected; the oldest(ffff880ffef2c000) timed out 1 secs ago, resid: 370304, wmem: 652800 Mar 5 10:03:49 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:03:49 dc2oss11 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 10:03:52 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 10:03:52 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 10:03:56 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002f_UUID' is not available for connect (no target) Mar 5 10:03:56 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 10:04:24 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:04:24 dc2oss07 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 10:04:24 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394031864 last 1394031864 Mar 5 10:04:24 dc2oss07 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 10:04:19 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:04:19 dc2oss01 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 10:04:19 dc2oss01 kernel: : Lustre: 15412:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394031859 last 1394031859 Mar 5 10:04:19 dc2oss01 kernel: : Lustre: 15412:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 10:04:33 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:04:33 dc2oss14 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 10:04:42 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 10:04:42 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:04:42 dc2oss13 kernel: : Lustre: dc2-OST0081: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) reconnecting Mar 5 10:04:42 dc2oss13 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 10:04:49 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:04:49 dc2oss09 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 10:04:53 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0071_UUID' is not available for connect (no target) Mar 5 10:04:53 dc2oss12 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 10:04:53 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:04:54 dc2oss10 kernel: : LustreError: 8779:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd0724800 x1461680445217981/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 0 to 0 dl 1394031918 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:04:54 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:04:54 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828036] Lustre: 23727:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669104679376 sent from dc2-OST0028-osc-ffff880956851000 to NID 10.10.0.4@o2ib 8s ago has timed out (8s prior to deadline). Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828043] Lustre: 23727:0:(client.c:1492:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828054] Lustre: dc2-OST0028-osc-ffff880956851000: Connection to service dc2-OST0028 via nid 10.10.0.4@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828056] Lustre: Skipped 1 previous similar message Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828417] Lustre: dc2-OST0028-osc-ffff880956851000: Connection restored to service dc2-OST0028 using nid 10.10.0.4@o2ib. Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828421] Lustre: Skipped 1 previous similar message Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828431] Lustre: Server dc2-OST0028_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 10:05:05 149.165.238.1 kernel: kernel: [4306277.828433] Lustre: Skipped 1 previous similar message Mar 5 10:04:57 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 26 secs ago Mar 5 10:04:57 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 26 secs ago Mar 5 10:05:33 dc2oss09 kernel: : Lustre: dc2-OST005e: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 10:05:33 dc2oss09 kernel: : LustreError: 14640:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881809609850 x1461680445218028/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394031996 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:05:33 dc2oss09 kernel: : LustreError: 14640:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 10:05:33 dc2oss09 kernel: : Lustre: dc2-OST005e: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:05:33 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:05:42 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:05:42 dc2oss15 kernel: : Lustre: Skipped 135 previous similar messages Mar 5 10:06:20 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0024_UUID' is not available for connect (no target) Mar 5 10:06:20 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:06:25 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:06:25 dc2oss06 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 10:06:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 10:06:42 dc2oss09 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:06:50 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 10:06:50 dc2oss13 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 10:06:53 dc2oss05 kernel: : Lustre: dc2-OST002e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:06:53 dc2oss05 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 10:07:02 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 10:07:02 dc2oss05 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:07:14 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:07:14 dc2oss12 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 10:07:32 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 10:07:32 dc2oss02 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 10:08:00 dc2oss12 kernel: : Lustre: dc2-OST0076: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 10:08:00 dc2oss12 kernel: : LustreError: 9289:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd04d7000 x1461679862997415/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032114 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:08:00 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 10:08:01 dc2oss12 kernel: : LustreError: 9335:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800b29d3c00 x1461679862997405/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032114 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:08:01 dc2oss12 kernel: : Lustre: dc2-OST0076: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 10:08:03 dc2oss10 kernel: : Lustre: dc2-OST0064: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:08:03 dc2oss10 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 10:08:16 dc2oss13 kernel: : Lustre: 5773:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394032089/real 1394032089] req@ffff880835273000 x1456766140433103/t0(0) o105->dc2-OST0080@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394032096 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 10:08:16 dc2oss13 kernel: : Lustre: 5773:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Mar 5 10:08:16 dc2oss13 kernel: : LustreError: 138-a: dc2-OST0080: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 10:08:17 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 10:08:17 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 10:09:03 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 10:09:03 dc2mds01 kernel: : Lustre: Skipped 47 previous similar messages Mar 5 10:09:03 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394032143 last 1394032143 Mar 5 10:09:03 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 49 previous similar messages Mar 5 10:09:21 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 10:09:21 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 10:09:21 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 10:09:21 dc2oss14 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:09:34 dc2oss11 kernel: : Lustre: dc2-OST006c: Client 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:09:34 dc2oss11 kernel: : LustreError: 25304:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88140358b850 x1461680445217977/t0(0) o3->6234af8c-f52f-8913-da83-114a3d3f52d4@149.165.229.55@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032646 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:09:34 dc2oss11 kernel: : Lustre: dc2-OST006c: Bulk IO read error with 6234af8c-f52f-8913-da83-114a3d3f52d4 (at 149.165.229.55@tcp), client will retry: rc -110 Mar 5 10:09:27 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 10:09:27 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 34 previous similar messages Mar 5 10:09:43 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x1db:0x0]] [valid 68719476736] Mar 5 10:09:43 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 9 previous similar messages Mar 5 10:10:01 dc2oss09 kernel: : Lustre: 16431:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25e45c00 cur 1394032201 last 1394032201 Mar 5 10:10:01 dc2oss09 kernel: : Lustre: 16431:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 145 previous similar messages Mar 5 10:10:07 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394032207 last 1394032207 Mar 5 10:10:07 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 148 previous similar messages Mar 5 10:10:14 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0040_UUID' is not available for connect (no target) Mar 5 10:10:14 dc2oss08 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 10:10:20 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 10:10:20 dc2oss07 kernel: : LustreError: Skipped 40 previous similar messages Mar 5 10:10:23 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394032223 last 1394032223 Mar 5 10:10:23 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 145 previous similar messages Mar 5 10:10:26 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1394032226 last 1394032226 Mar 5 10:10:26 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 10:10:58 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:10:58 dc2oss08 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 10:11:00 dc2oss11 kernel: : Lustre: Service thread pid 25413 was inactive for 494.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:11:00 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:11:00 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032260.25413 Mar 5 10:11:04 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394032264 last 1394032264 Mar 5 10:11:04 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 10:11:16 dc2oss11 kernel: : Lustre: Service thread pid 25433 was inactive for 494.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:11:16 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:11:16 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032276.25433 Mar 5 10:11:19 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394032279 last 1394032279 Mar 5 10:11:19 dc2oss14 kernel: : Lustre: 15783:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 10:11:25 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394032285 last 1394032285 Mar 5 10:11:25 dc2oss15 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 157 previous similar messages Mar 5 10:12:08 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:12:08 dc2oss03 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 10:12:08 dc2oss03 kernel: : Lustre: 16594:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394032328 last 1394032327 Mar 5 10:12:08 dc2oss03 kernel: : Lustre: 16594:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 10:12:05 dc2oss16 kernel: : Lustre: 17221:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394032325 last 1394032325 Mar 5 10:12:05 dc2oss16 kernel: : Lustre: 17221:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 10:12:10 dc2oss08 kernel: : Lustre: 16027:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394032323/real 1394032324] req@ffff880c70f8bc00 x1456766190323978/t0(0) o106->dc2-OST004b@149.165.229.111@tcp:15/16 lens 296/232 e 0 to 1 dl 1394032330 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 5 10:12:10 dc2oss08 kernel: : Lustre: 16027:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Mar 5 10:12:18 dc2oss06 kernel: : Lustre: 15765:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394032338 last 1394032338 Mar 5 10:12:18 dc2oss06 kernel: : Lustre: 15765:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 10:12:19 dc2oss11 kernel: : Lustre: Service thread pid 8808 was inactive for 494.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:12:19 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:12:19 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032339.8808 Mar 5 10:12:20 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:12:20 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032340.14660 Mar 5 10:12:31 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394032351 last 1394032351 Mar 5 10:12:31 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 158 previous similar messages Mar 5 10:12:41 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394032361 last 1394032361 Mar 5 10:12:41 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 10:12:56 gc1 kernel: : Lustre: 4798:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0x83:0x0]] [valid 68719476736] Mar 5 10:13:00 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:13:00 dc2oss02 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 10:13:00 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394032380 last 1394032380 Mar 5 10:13:00 dc2oss02 kernel: : Lustre: 15937:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 10:12:59 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394032379 last 1394032379 Mar 5 10:12:59 dc2oss05 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 10:13:01 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0016_UUID' is not available for connect (no target) Mar 5 10:13:01 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:13:04 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:13:04 dc2oss04 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 10:13:20 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:13:20 dc2oss16 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 10:13:53 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 10:13:53 dc2oss11 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 10:13:55 dc2oss10 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 110s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST0061_UUID lock: ffff881100c89240/0xef62ad91ae2216fe lrc: 3/0,0 mode: PR/PR res: 23279772/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c57823a47 expref: 223 pid: 16313 timeout 9047667840 Mar 5 10:14:07 dc2oss11 kernel: : Lustre: 8741:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-343), not sending early reply Mar 5 10:14:10 dc2oss11 kernel: : Lustre: 25458:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-343), not sending early reply Mar 5 10:14:12 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:12 dc2oss11 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 10:14:12 dc2oss11 kernel: : LustreError: 8820:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 948+0s req@ffff880c0e6fc400 x1442912236874483/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394032452 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:14:12 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:14:12 dc2oss11 kernel: : Lustre: Skipped 7 previous similar messages Mar 5 10:14:12 dc2oss11 kernel: : Lustre: Service thread pid 8820 completed after 948.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:14:12 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:14:15 dc2oss11 kernel: : LustreError: 14648:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 948+0s req@ffff8817faf67050 x1442912236874487/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394032455 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:14:15 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:14:15 dc2oss11 kernel: : Lustre: Service thread pid 14648 completed after 948.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:14:09 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0033_UUID' is not available for connect (no target) Mar 5 10:14:09 dc2oss06 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:14:12 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 528992, left: 528992, last alive is 4 secs ago Mar 5 10:14:12 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 528992, left: 528992, last alive is 4 secs ago Mar 5 10:14:15 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 10:14:15 dc2oss10 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:14:19 dc2oss11 kernel: : Lustre: 25353:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-343), not sending early reply Mar 5 10:14:24 dc2oss11 kernel: : LustreError: 14703:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 948+0s req@ffff8804065fe800 x1442912236874495/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394032464 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:14:24 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:14:24 dc2oss11 kernel: : Lustre: Service thread pid 14703 completed after 948.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:14:34 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:34 dc2oss07 kernel: : Lustre: Skipped 138 previous similar messages Mar 5 10:14:34 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394032474 last 1394032474 Mar 5 10:14:34 dc2oss07 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 148 previous similar messages Mar 5 10:14:35 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:35 dc2oss14 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 10:14:34 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:34 dc2oss01 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 10:14:34 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394032474 last 1394032474 Mar 5 10:14:34 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 10:14:44 dc2oss13 kernel: : Lustre: dc2-OST0085: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:44 dc2oss13 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 10:14:54 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0069_UUID' is not available for connect (no target) Mar 5 10:14:54 dc2oss12 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:14:59 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:14:59 dc2oss09 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 10:15:04 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 10:15:04 dc2oss15 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 10:15:49 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:15:49 dc2oss15 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 10:15:58 dc2oss11 kernel: : Lustre: 25345:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-192), not sending early reply Mar 5 10:16:03 dc2oss11 kernel: : LustreError: 25413:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 797+0s req@ffff8817fa4af850 x1442912236877518/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394032563 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:16:03 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:16:03 dc2oss11 kernel: : Lustre: Service thread pid 25413 completed after 797.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:16:30 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 10:16:30 dc2oss03 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 10:16:14 dc2oss11 kernel: : Lustre: 14718:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-192), not sending early reply Mar 5 10:16:19 dc2oss11 kernel: : LustreError: 25433:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 797+0s req@ffff8800afa81000 x1442912236877741/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394032579 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:16:19 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:16:19 dc2oss11 kernel: : Lustre: Service thread pid 25433 completed after 797.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:16:25 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:16:25 dc2oss06 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 10:16:26 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:16:26 dc2oss11 kernel: : LustreError: 25301:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd7350c00 x1442912236883654/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394033013 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:16:26 dc2oss11 kernel: : LustreError: 25301:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 10:16:27 dc2oss11 kernel: : Lustre: Service thread pid 8808 completed after 742.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:16:54 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:16:54 dc2oss05 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 10:17:17 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 10:17:17 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 10:17:41 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 10:17:41 dc2oss02 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:17:31 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:17:31 dc2oss12 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 10:17:35 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 10:17:35 dc2oss13 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 10:17:37 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 10:17:37 dc2oss09 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 10:17:51 dc2oss05 kernel: : Lustre: dc2-OST002d: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:17:52 dc2oss05 kernel: : LustreError: 29843:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8801b670c400 x1461679898900522/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032713 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:17:52 dc2oss05 kernel: : LustreError: 29843:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:17:52 dc2oss05 kernel: : Lustre: dc2-OST002d: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:17:52 dc2oss05 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:18:02 dc2oss14 kernel: : Lustre: dc2-OST0090: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 10:18:02 dc2oss14 kernel: : LustreError: 22702:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880befee2050 x1461679898901022/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032717 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:18:02 dc2oss14 kernel: : Lustre: dc2-OST0090: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:18:02 dc2oss14 kernel: : LustreError: 22680:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc7a29000 x1461679898901015/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032716 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:18:02 dc2oss14 kernel: : Lustre: dc2-OST0090: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:18:04 dc2oss10 kernel: : Lustre: dc2-OST0067: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 10:18:04 dc2oss10 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 10:18:35 dc2oss02 kernel: : Lustre: dc2-OST000e: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 10:18:35 dc2oss02 kernel: : LustreError: 20654:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88078e6da400 x1461679898901019/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032793 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:18:35 dc2oss02 kernel: : Lustre: dc2-OST000e: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:18:36 dc2oss02 kernel: : LustreError: 21042:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880792268400 x1461679898901009/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032793 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:18:36 dc2oss02 kernel: : Lustre: dc2-OST000e: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:18:32 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 10:18:32 dc2oss01 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:19:03 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:19:03 dc2oss04 kernel: : LustreError: 20060:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88181137e050 x1461679898900533/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032771 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:19:03 dc2oss04 kernel: : LustreError: 20060:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:19:03 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:19:03 dc2oss04 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:19:41 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 10:19:41 dc2oss14 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:19:44 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 10:19:44 dc2oss16 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:20:09 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394032809 last 1394032809 Mar 5 10:20:09 dc2oss12 kernel: : Lustre: 14443:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 10:20:04 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394032804 last 1394032804 Mar 5 10:20:04 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 10:20:13 dc2oss14 kernel: : Lustre: Service thread pid 22792 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:20:13 dc2oss14 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:20:13 dc2oss14 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:20:13 dc2oss14 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:20:13 dc2oss14 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032813.22792 Mar 5 10:20:20 dc2oss14 kernel: : Lustre: Service thread pid 32107 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:20:20 dc2oss14 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:20:20 dc2oss14 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:20:20 dc2oss14 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:20:20 dc2oss14 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032820.32107 Mar 5 10:20:24 dc2oss02 kernel: : Lustre: Service thread pid 21122 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:20:24 dc2oss02 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:20:24 dc2oss02 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:20:24 dc2oss02 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:20:24 dc2oss02 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032824.21122 Mar 5 10:20:26 dc2oss02 kernel: : Lustre: Service thread pid 20288 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:20:26 dc2oss02 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:20:26 dc2oss02 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:20:26 dc2oss02 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:20:26 dc2oss02 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032826.20288 Mar 5 10:20:15 dc2oss10 kernel: : Lustre: Service thread pid 15952 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:20:15 dc2oss10 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:20:15 dc2oss10 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:20:15 dc2oss10 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:20:15 dc2oss10 kernel: : LustreError: dumping log to /tmp/lustre-log.1394032815.15952 Mar 5 10:20:25 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394032825 last 1394032825 Mar 5 10:20:25 dc2oss13 kernel: : Lustre: 15800:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 10:20:24 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:20:24 dc2mds01 kernel: : Lustre: Skipped 25 previous similar messages Mar 5 10:20:24 dc2mds01 kernel: : Lustre: 1376:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394032824 last 1394032824 Mar 5 10:20:24 dc2mds01 kernel: : Lustre: 1376:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 25 previous similar messages Mar 5 10:20:44 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0053: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37520147 exp ffff880c04430000 cur 1394032844 last 1394032844 Mar 5 10:20:44 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 10:21:00 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 10:21:00 dc2oss07 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 10:21:06 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 10:21:06 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:21:15 dc2oss08 kernel: : Lustre: dc2-OST004c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) reconnecting Mar 5 10:21:15 dc2oss08 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 10:21:32 dc2oss03 kernel: : Lustre: dc2-OST001c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:21:32 dc2oss03 kernel: : LustreError: 16745:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88180572c850 x1461680520713218/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032931 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:21:32 dc2oss03 kernel: : Lustre: dc2-OST001c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:21:08 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp t0 exp ffff88153dfa7400 cur 1394032868 last 1394032868 Mar 5 10:21:08 dc2oss04 kernel: : Lustre: 16494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 10:21:12 dc2oss12 kernel: : Lustre: dc2-OST007c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:21:13 dc2oss12 kernel: : LustreError: 32730:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd0fa2c00 x1461680520713261/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032907 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:21:13 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:21:13 dc2oss12 kernel: : LustreError: 9160:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8802c04b8000 x1461680520713199/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032905 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:21:13 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:21:17 dc2oss06 kernel: : Lustre: dc2-OST0039: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:21:18 dc2oss06 kernel: : LustreError: 6874:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800ab24ec00 x1461680520713243/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394032907 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:21:18 dc2oss06 kernel: : LustreError: 6874:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 7 previous similar messages Mar 5 10:21:18 dc2oss06 kernel: : Lustre: dc2-OST0039: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:21:18 dc2oss06 kernel: : Lustre: Skipped 7 previous similar messages Mar 5 10:21:25 dc2oss15 kernel: : Lustre: 15654:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394032885 last 1394032885 Mar 5 10:21:25 dc2oss15 kernel: : Lustre: 15654:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 10:21:33 dc2oss14 kernel: : Lustre: 15638:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp t0 exp ffff880beb427800 cur 1394032893 last 1394032893 Mar 5 10:21:33 dc2oss14 kernel: : Lustre: 15638:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 10:21:22 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 10:21:22 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 16 previous similar messages Mar 5 10:21:28 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x8d4c:0x0]] [valid 68719476736] Mar 5 10:21:28 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 9 previous similar messages Mar 5 10:22:24 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:22:24 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 10:22:24 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394032944 last 1394032944 Mar 5 10:22:24 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 10:22:05 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394032925 last 1394032925 Mar 5 10:22:05 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 133 previous similar messages Mar 5 10:22:18 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1394032938 last 1394032938 Mar 5 10:22:18 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 10:22:45 dc2oss10 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394032965 last 1394032965 Mar 5 10:22:45 dc2oss10 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 10:22:45 dc2oss04 kernel: : Lustre: dc2-OST0028: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 10:22:46 dc2oss04 kernel: : LustreError: 16157:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b0c049c00 x1461680520713267/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394033077 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:22:46 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:22:50 dc2oss07 kernel: : Lustre: dc2-OST0041: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 10:22:50 dc2oss07 kernel: : LustreError: 14484:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf8453850 x1461680520713254/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 2 to 0 dl 1394033023 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:22:50 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:22:50 dc2oss07 kernel: : LustreError: 6201:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bdd0c0400 x1461680520713227/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 2 to 0 dl 1394033021 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:22:50 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:23:02 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:23:02 dc2oss02 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 10:23:02 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394032982 last 1394032982 Mar 5 10:23:02 dc2oss02 kernel: : Lustre: 15882:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 10:22:56 dc2oss11 kernel: : Lustre: 16224:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394032976 last 1394032976 Mar 5 10:22:56 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394032976 last 1394032976 Mar 5 10:22:56 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 10:23:00 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394032979 last 1394032979 Mar 5 10:23:00 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 10:23:03 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 10:23:03 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:23:06 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:23:06 dc2oss04 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 10:23:22 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:23:22 dc2oss16 kernel: : Lustre: Skipped 143 previous similar messages Mar 5 10:23:20 dc2oss01 kernel: : Lustre: Service thread pid 15124 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033000.15124 Mar 5 10:23:20 dc2oss01 kernel: : Lustre: Service thread pid 19586 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033000.19586 Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:23:20 dc2oss01 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033000.15176 Mar 5 10:23:31 dc2oss06 kernel: : Lustre: Service thread pid 7069 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:23:31 dc2oss06 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:23:31 dc2oss06 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:23:31 dc2oss06 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:23:31 dc2oss06 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033011.7069 Mar 5 10:24:01 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 10:24:01 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 10:24:24 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 10:24:24 dc2oss10 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 10:24:15 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 10:24:15 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 10:24:19 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:24:19 dc2oss11 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 10:24:38 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:24:38 dc2oss07 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 10:24:38 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394033078 last 1394033078 Mar 5 10:24:38 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 10:24:49 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:24:49 dc2oss13 kernel: : Lustre: Skipped 137 previous similar messages Mar 5 10:24:59 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:24:59 dc2oss14 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 10:24:50 dc2oss01 kernel: : Lustre: dc2-OST0009: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:24:50 dc2oss01 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 10:24:50 dc2oss01 kernel: : Lustre: 15412:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be3acd800 cur 1394033090 last 1394033090 Mar 5 10:24:50 dc2oss01 kernel: : Lustre: 15412:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 10:25:20 dc2oss03 kernel: : Lustre: Service thread pid 21184 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:25:20 dc2oss03 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 10:25:20 dc2oss03 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 10:25:20 dc2oss03 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:25:20 dc2oss03 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033120.21184 Mar 5 10:25:07 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:25:07 dc2oss09 kernel: : Lustre: Skipped 143 previous similar messages Mar 5 10:25:34 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 10:25:34 dc2oss12 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:25:49 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:25:49 dc2oss15 kernel: : Lustre: Skipped 136 previous similar messages Mar 5 10:26:05 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 10:26:05 dc2oss15 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 10:26:39 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:26:39 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:26:39 dc2oss06 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 10:26:39 dc2oss06 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 10:26:48 dc2oss08 kernel: : Lustre: 16148:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394033199/real 1394033199] req@ffff8817ec5ed400 x1456766190327466/t0(0) o104->dc2-OST0052@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394033208 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 10:26:48 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0052: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 10:26:51 dc2oss08 kernel: : LustreError: 26701:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.226.206@tcp arrived at 1394033211 with bad export cookie 698067220715156104 Mar 5 10:26:58 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:26:58 dc2oss05 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 10:27:21 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 10:27:21 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 10:27:46 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 10:27:46 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 10:27:42 dc2oss12 kernel: : Lustre: dc2-OST0075: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:27:42 dc2oss12 kernel: : Lustre: Skipped 127 previous similar messages Mar 5 10:27:40 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 593120, left: 593120, last alive is 27 secs ago Mar 5 10:27:40 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 593120, left: 593120, last alive is 27 secs ago Mar 5 10:27:42 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0067_UUID' is not available for connect (no target) Mar 5 10:27:42 dc2oss09 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:28:10 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 10:28:10 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:28:23 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:28:23 dc2oss10 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 10:28:47 dc2oss02 kernel: : Lustre: 21074:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-103), not sending early reply Mar 5 10:28:47 dc2oss02 kernel: : Lustre: 21074:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Mar 5 10:28:49 dc2oss02 kernel: : Lustre: 20929:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-103), not sending early reply Mar 5 10:28:52 dc2oss02 kernel: : LustreError: 21122:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 708+0s req@ffff880708240400 x1461679898900529/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 3 to 0 dl 1394033332 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:28:52 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:28:52 dc2oss02 kernel: : Lustre: Service thread pid 21122 completed after 708.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:28:54 dc2oss02 kernel: : LustreError: 20288:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 708+0s req@ffff88078eb9c000 x1461679898900981/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 3 to 0 dl 1394033334 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:28:54 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:28:54 dc2oss02 kernel: : Lustre: Service thread pid 20288 completed after 708.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:28:36 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 10:28:36 dc2oss01 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 10:29:11 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 10:29:11 dc2oss03 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 10:29:49 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 10:29:49 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:29:53 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 10:29:53 dc2oss14 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:30:02 dc2oss11 kernel: : Lustre: Service thread pid 14652 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:30:02 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:30:02 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:30:02 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033402.14652 Mar 5 10:30:08 dc2oss11 kernel: : Lustre: Service thread pid 8721 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:30:08 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:30:08 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033408.8721 Mar 5 10:30:05 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394033405 last 1394033405 Mar 5 10:30:05 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 10:30:17 dc2oss12 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394033417 last 1394033417 Mar 5 10:30:17 dc2oss12 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 10:30:24 dc2oss11 kernel: : Lustre: Service thread pid 14623 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:30:24 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:30:24 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394033424.14623 Mar 5 10:30:26 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394033426 last 1394033426 Mar 5 10:30:26 dc2oss13 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 10:30:38 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 10:30:38 dc2mds01 kernel: : Lustre: Skipped 19 previous similar messages Mar 5 10:30:38 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394033438 last 1394033438 Mar 5 10:30:38 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 19 previous similar messages Mar 5 10:30:31 dc2oss01 kernel: : Lustre: 15166:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-31), not sending early reply Mar 5 10:30:36 dc2oss01 kernel: : LustreError: 15124:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 636+0s req@ffff8806e4fc9c00 x1461680520713205/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394033436 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:30:36 dc2oss01 kernel: : Lustre: dc2-OST0005: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:30:36 dc2oss01 kernel: : Lustre: Service thread pid 15124 completed after 636.05s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:30:36 dc2oss01 kernel: : LustreError: 19586:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 636+0s req@ffff880c25fd0800 x1461680520713233/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394033436 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:30:36 dc2oss01 kernel: : Lustre: dc2-OST0005: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:30:36 dc2oss01 kernel: : Lustre: Service thread pid 19586 completed after 636.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:31:14 dc2oss08 kernel: : Lustre: 15923:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1394033474 last 1394033474 Mar 5 10:31:14 dc2oss08 kernel: : Lustre: 16028:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394033474 last 1394033474 Mar 5 10:31:14 dc2oss08 kernel: : Lustre: 16028:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 61 previous similar messages Mar 5 10:31:18 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:31:18 dc2oss08 kernel: : Lustre: Skipped 62 previous similar messages Mar 5 10:31:11 dc2oss04 kernel: : Lustre: 16451:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394033471 last 1394033471 Mar 5 10:31:11 dc2oss04 kernel: : Lustre: 16451:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 10:31:21 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 10:31:21 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 10:31:27 dc2oss15 kernel: : Lustre: 16348:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394033487 last 1394033487 Mar 5 10:31:27 dc2oss15 kernel: : Lustre: 16348:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 10:31:29 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 10:31:29 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:31:38 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37972127 exp ffff880c08fefc00 cur 1394033498 last 1394033498 Mar 5 10:31:38 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 10:31:33 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 10:31:33 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 12 previous similar messages Mar 5 10:31:56 dc2oss14 kernel: : Lustre: 22726:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-303), not sending early reply Mar 5 10:32:01 dc2oss14 kernel: : LustreError: 22792:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 908+0s req@ffff880bcc583400 x1461679898900520/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 4 to 0 dl 1394033521 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:32:01 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:32:01 dc2oss14 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 10:32:01 dc2oss14 kernel: : Lustre: Service thread pid 22792 completed after 908.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:32:03 dc2oss14 kernel: : Lustre: 22641:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-303), not sending early reply Mar 5 10:31:58 dc2oss10 kernel: : Lustre: 8860:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-303), not sending early reply Mar 5 10:32:03 dc2oss10 kernel: : LustreError: 15952:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 908+0s req@ffff880bbc433800 x1461679898900758/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 4 to 0 dl 1394033523 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:32:03 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:32:03 dc2oss10 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 10:32:03 dc2oss10 kernel: : Lustre: Service thread pid 15952 completed after 908.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:31:50 dc2mds01 kernel: : Lustre: 14543:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f9fb:0x8ee5:0x0]] [valid 68719476736] Mar 5 10:31:50 dc2mds01 kernel: : Lustre: 14543:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 29 previous similar messages Mar 5 10:32:06 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394033526 last 1394033526 Mar 5 10:32:06 dc2oss16 kernel: : Lustre: 17293:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 86 previous similar messages Mar 5 10:32:26 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:32:26 dc2oss03 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 10:32:26 dc2oss03 kernel: : Lustre: 17489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394033546 last 1394033546 Mar 5 10:32:26 dc2oss03 kernel: : Lustre: 17489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 10:32:08 dc2oss14 kernel: : LustreError: 32107:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 908+0s req@ffff880bc732bc00 x1461679898901013/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 4 to 0 dl 1394033528 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:32:08 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 10:32:08 dc2oss14 kernel: : Lustre: Service thread pid 32107 completed after 908.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:32:26 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394033546 last 1394033546 Mar 5 10:32:26 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 10:32:46 dc2oss10 kernel: : Lustre: 16566:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394033566 last 1394033566 Mar 5 10:32:46 dc2oss10 kernel: : Lustre: 16566:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 10:33:09 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:33:09 dc2oss02 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 10:33:09 dc2oss02 kernel: : Lustre: 16418:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394033589 last 1394033589 Mar 5 10:33:09 dc2oss02 kernel: : Lustre: 16418:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 10:33:04 dc2oss05 kernel: : Lustre: 15813:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394033584 last 1394033584 Mar 5 10:33:04 dc2oss05 kernel: : Lustre: 14589:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fae4a000 cur 1394033584 last 1394033584 Mar 5 10:33:04 dc2oss05 kernel: : Lustre: 14589:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 72 previous similar messages Mar 5 10:33:17 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394033597 last 1394033597 Mar 5 10:33:17 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 10:33:06 dc2mds01 kernel: : LustreError: 9828:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232059765513, new: 232059765512 replay: 0. see LU-617. Mar 5 10:33:15 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:33:15 dc2oss04 kernel: : Lustre: Skipped 70 previous similar messages Mar 5 10:33:23 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:33:23 dc2oss16 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 10:34:02 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 10:34:02 dc2oss04 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 10:34:04 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 10:34:04 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 10:34:20 dc2oss11 kernel: : Lustre: dc2-OST0069: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:34:20 dc2oss11 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 10:34:37 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 10:34:37 dc2oss10 kernel: : LustreError: Skipped 12 previous similar messages Mar 5 10:34:36 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 376672, left: 376672, last alive is 8 secs ago Mar 5 10:34:36 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 376672, left: 376672, last alive is 8 secs ago Mar 5 10:34:52 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:34:52 dc2oss13 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 10:34:59 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:34:59 dc2oss14 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 10:34:54 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:34:54 dc2oss01 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 10:34:54 dc2oss01 kernel: : Lustre: 15021:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394033694 last 1394033694 Mar 5 10:34:54 dc2oss01 kernel: : Lustre: 15021:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 10:35:15 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:35:15 dc2oss07 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 10:35:15 dc2oss07 kernel: : Lustre: 15907:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394033715 last 1394033715 Mar 5 10:35:15 dc2oss07 kernel: : Lustre: 15907:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 10:35:14 dc2oss06 kernel: : Lustre: 14577:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-303), not sending early reply Mar 5 10:35:14 dc2oss06 kernel: : Lustre: 14577:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Mar 5 10:35:19 dc2oss06 kernel: : LustreError: 7069:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 908+0s req@ffff880bd1894c00 x1461680520713263/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 4 to 0 dl 1394033719 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:35:19 dc2oss06 kernel: : LustreError: 7069:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:35:19 dc2oss06 kernel: : Lustre: dc2-OST003d: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:35:19 dc2oss06 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 10:35:19 dc2oss06 kernel: : Lustre: Service thread pid 7069 completed after 908.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:35:19 dc2oss06 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:35:20 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:35:20 dc2oss09 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 10:35:43 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 10:35:43 dc2oss12 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 10:35:50 dc2oss15 kernel: : Lustre: dc2-OST0097: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:35:50 dc2oss15 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 10:36:22 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 10:36:22 dc2oss15 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 10:36:21 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0031_UUID' is not available for connect (no target) Mar 5 10:36:21 dc2oss06 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:36:41 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:36:41 dc2oss06 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 10:37:03 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:37:03 dc2oss05 kernel: : Lustre: Skipped 72 previous similar messages Mar 5 10:37:51 dc2oss03 kernel: : Lustre: 20868:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-351), not sending early reply Mar 5 10:37:56 dc2oss03 kernel: : LustreError: 21184:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 956+0s req@ffff880651aac800 x1461680520714215/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 4 to 0 dl 1394033876 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 10:37:56 dc2oss03 kernel: : Lustre: dc2-OST001c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 10:37:56 dc2oss03 kernel: : Lustre: Service thread pid 21184 completed after 956.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:37:32 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 10:37:32 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 10:37:52 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:37:52 dc2oss12 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 10:38:03 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 10:38:03 dc2oss09 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:38:29 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0089_UUID' is not available for connect (no target) Mar 5 10:38:29 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:38:32 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:38:32 dc2oss10 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 10:38:41 dc2oss11 kernel: : Lustre: 8657:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-119), not sending early reply Mar 5 10:38:46 dc2oss11 kernel: : LustreError: 14652:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 724+0s req@ffff8800796ed400 x1442912236899351/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394033926 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:38:46 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:38:46 dc2oss11 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 10:38:46 dc2oss11 kernel: : Lustre: Service thread pid 14652 completed after 724.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:38:46 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 10:39:02 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 10:39:02 dc2oss02 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 10:38:50 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 10:38:50 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 10:39:03 dc2oss11 kernel: : Lustre: 14684:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-135), not sending early reply Mar 5 10:39:08 dc2oss11 kernel: : LustreError: 8721:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 740+0s req@ffff880bd3a85c00 x1442912236899355/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394033948 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:39:08 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:39:08 dc2oss11 kernel: : Lustre: Service thread pid 8721 completed after 740.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:39:15 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 833632, left: 833632, last alive is 38 secs ago Mar 5 10:39:15 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 833632, left: 833632, last alive is 38 secs ago Mar 5 10:39:19 dc2oss11 kernel: : Lustre: 25345:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-135), not sending early reply Mar 5 10:39:24 dc2oss11 kernel: : LustreError: 14623:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 740+0s req@ffff880bf3853400 x1442912236899586/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394033964 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:39:24 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:39:24 dc2oss11 kernel: : Lustre: Service thread pid 14623 completed after 740.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 10:39:46 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 10:39:46 dc2oss03 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:39:53 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 206432, left: 206432, last alive is 37 secs ago Mar 5 10:39:53 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 206432, left: 206432, last alive is 37 secs ago Mar 5 10:39:59 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 10:39:59 dc2oss16 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:40:02 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 10:40:02 dc2oss14 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 10:40:06 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806d9c400 cur 1394034006 last 1394034006 Mar 5 10:40:06 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 10:40:31 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 10:40:31 dc2oss11 kernel: : LustreError: 25451:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd68a0c00 x1442912236911303/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394034785 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:40:31 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 10:40:31 dc2oss11 kernel: : LustreError: 25451:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 10:40:32 dc2oss11 kernel: : LustreError: 25395:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880146f9f400 x1442912236910174/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394034390 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:40:32 dc2oss11 kernel: : LustreError: 8633:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc6c06800 x1442912236907364/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394034111 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:40:32 dc2oss11 kernel: : LustreError: 8633:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 10:40:26 dc2oss12 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394034026 last 1394034026 Mar 5 10:40:26 dc2oss12 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 10:40:31 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394034031 last 1394034031 Mar 5 10:40:31 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 10:40:33 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 10:40:33 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 10:40:33 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 1 previous similar message Mar 5 10:40:33 dc2xfer03 kernel: : Lustre: 2644:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) Skipped 1 previous similar message Mar 5 10:40:34 dc2xfer01 kernel: : Lustre: 2780:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 10:40:34 dc2xfer01 kernel: : Lustre: 2780:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 10:40:39 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 10:40:39 dc2mds01 kernel: : Lustre: Skipped 45 previous similar messages Mar 5 10:40:39 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394034039 last 1394034039 Mar 5 10:40:39 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 45 previous similar messages Mar 5 10:41:11 dc2oss04 kernel: : Lustre: 16682:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from b1e33545-824c-4da6-8861-050a201417c1@149.165.229.84@tcp t0 exp ffff88145c985800 cur 1394034071 last 1394034071 Mar 5 10:41:11 dc2oss04 kernel: : Lustre: 16682:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 10:41:31 dc2oss15 kernel: : Lustre: 15828:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394034091 last 1394034091 Mar 5 10:41:31 dc2oss15 kernel: : Lustre: 15828:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 10:41:36 dc2oss15 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 110s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST009c_UUID lock: ffff8813f651a6c0/0x915460122ba86ba7 lrc: 3/0,0 mode: PW/PW res: 23152881/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 remote: 0xfd05479c579716ec expref: 320 pid: 15772 timeout 9049312373 Mar 5 10:41:36 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:41:36 dc2oss08 kernel: : Lustre: Skipped 71 previous similar messages Mar 5 10:41:36 dc2oss08 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394034096 last 1394034096 Mar 5 10:41:36 dc2oss08 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 10:41:39 dc2oss14 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394034099 last 1394034099 Mar 5 10:41:39 dc2oss14 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 10:41:53 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 10:41:53 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 10:41:54 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 10:41:54 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 33 previous similar messages Mar 5 10:42:07 dc2oss16 kernel: : Lustre: 17397:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05c45400 cur 1394034127 last 1394034127 Mar 5 10:42:07 dc2oss16 kernel: : Lustre: 17397:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 10:42:44 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:42:44 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 10:42:44 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0df95000 cur 1394034164 last 1394034164 Mar 5 10:42:44 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 10:42:14 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb08:0x1:0x0]] [valid 68719476736] Mar 5 10:42:14 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 35 previous similar messages Mar 5 10:42:27 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394034147 last 1394034147 Mar 5 10:42:27 dc2oss06 kernel: : Lustre: 15773:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 10:42:58 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394034178 last 1394034178 Mar 5 10:42:58 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 10:43:19 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394034199 last 1394034199 Mar 5 10:43:19 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 10:43:31 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:43:31 dc2oss02 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 10:43:31 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394034211 last 1394034211 Mar 5 10:43:31 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 10:43:26 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:43:26 dc2oss16 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 10:43:22 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:43:22 dc2oss04 kernel: : Lustre: Skipped 130 previous similar messages Mar 5 10:43:49 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394034229 last 1394034229 Mar 5 10:43:49 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 10:44:12 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 10:44:12 dc2oss04 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 10:44:20 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:44:20 dc2oss11 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 10:44:20 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 10:44:20 dc2oss11 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 10:44:38 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 10:44:38 dc2oss07 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 10:44:41 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 10:44:41 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 10:44:59 dc2oss13 kernel: : Lustre: dc2-OST0080: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:44:59 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 10:45:02 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:45:02 dc2oss14 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 10:44:55 dc2oss01 kernel: : Lustre: dc2-OST0006: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:44:55 dc2oss01 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 10:44:55 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d8f0400 cur 1394034295 last 1394034295 Mar 5 10:44:55 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 10:45:29 dc2oss09 kernel: : Lustre: dc2-OST0056: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:45:29 dc2oss09 kernel: : Lustre: Skipped 133 previous similar messages Mar 5 10:45:56 dc2oss15 kernel: : Lustre: dc2-OST0097: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:45:56 dc2oss15 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 10:46:02 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 10:46:02 dc2oss12 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 10:46:34 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 10:46:34 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 10:46:42 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:46:42 dc2oss06 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 10:46:50 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 10:46:50 dc2oss06 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 10:47:04 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:47:04 dc2oss05 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 10:47:05 dc2oss07 kernel: : Lustre: dc2-OST0046: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:47:05 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:47:05 dc2oss07 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 10:47:05 dc2oss07 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 10:47:05 dc2oss07 kernel: : Lustre: 15962:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394034425 last 1394034425 Mar 5 10:47:05 dc2oss07 kernel: : Lustre: 15962:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 10:47:34 149.165.238.1 kernel: kernel: [4308826.504123] Lustre: 6305:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105100468 sent from dc2-OST002f-osc-ffff880956851000 to NID 10.10.0.5@o2ib 8s ago has timed out (8s prior to deadline). Mar 5 10:47:34 149.165.238.1 kernel: kernel: [4308826.504136] Lustre: dc2-OST002f-osc-ffff880956851000: Connection to service dc2-OST002f via nid 10.10.0.5@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 10:47:34 149.165.238.1 kernel: kernel: [4308826.504486] Lustre: dc2-OST002f-osc-ffff880956851000: Connection restored to service dc2-OST002f using nid 10.10.0.5@o2ib. Mar 5 10:47:34 149.165.238.1 kernel: kernel: [4308826.504507] Lustre: Server dc2-OST002f_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 10:47:55 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:47:55 dc2oss12 kernel: : Lustre: Skipped 69 previous similar messages Mar 5 10:48:14 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 10:48:14 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 10:48:30 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 10:48:30 dc2oss05 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 10:48:40 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:48:40 dc2oss10 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 10:49:08 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 10:49:08 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 10:48:54 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 10:48:54 dc2oss01 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 10:49:12 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008b_UUID' is not available for connect (no target) Mar 5 10:49:12 dc2oss13 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 10:50:17 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394034617 last 1394034617 Mar 5 10:50:17 dc2oss09 kernel: : Lustre: 16243:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 10:50:25 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 10:50:25 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 10:50:29 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009c_UUID' is not available for connect (no target) Mar 5 10:50:29 dc2oss16 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 10:50:29 dc2oss12 kernel: : Lustre: 15392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394034629 last 1394034629 Mar 5 10:50:29 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394034629 last 1394034629 Mar 5 10:50:29 dc2oss12 kernel: : Lustre: 16088:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 10:50:38 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394034638 last 1394034638 Mar 5 10:50:38 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 10:50:40 dc2oss15 kernel: : Lustre: dc2-OST009b: Client 1999ba24-fd4d-a22a-36d0-22645de31e28 (at 149.165.229.23@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 10:50:40 dc2oss15 kernel: : LustreError: 8407:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd0a96000 x1461681515221462/t0(0) o3->1999ba24-fd4d-a22a-36d0-22645de31e28@149.165.229.23@tcp:0/0 lens 448/400 e 0 to 0 dl 1394034646 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:50:40 dc2oss15 kernel: : LustreError: 8407:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 10:50:40 dc2oss15 kernel: : Lustre: dc2-OST009b: Bulk IO read error with 1999ba24-fd4d-a22a-36d0-22645de31e28 (at 149.165.229.23@tcp), client will retry: rc -110 Mar 5 10:50:40 dc2oss15 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 10:50:41 dc2oss15 kernel: : LustreError: 25609:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8818079c4050 x1461681515221435/t0(0) o3->1999ba24-fd4d-a22a-36d0-22645de31e28@149.165.229.23@tcp:0/0 lens 448/400 e 1 to 0 dl 1394034684 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 10:50:41 dc2oss15 kernel: : Lustre: dc2-OST009b: Bulk IO read error with 1999ba24-fd4d-a22a-36d0-22645de31e28 (at 149.165.229.23@tcp), client will retry: rc -110 Mar 5 10:50:39 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 10:50:39 dc2mds01 kernel: : Lustre: Skipped 20 previous similar messages Mar 5 10:50:39 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394034639 last 1394034639 Mar 5 10:50:39 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 22 previous similar messages Mar 5 10:51:38 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 10:51:38 dc2oss03 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:51:20 dc2oss04 kernel: : Lustre: 16759:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394034680 last 1394034680 Mar 5 10:51:20 dc2oss04 kernel: : Lustre: 16759:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 71 previous similar messages Mar 5 10:51:35 dc2oss15 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394034695 last 1394034695 Mar 5 10:51:35 dc2oss15 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 10:51:40 dc2oss14 kernel: : Lustre: 15475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394034700 last 1394034700 Mar 5 10:51:40 dc2oss14 kernel: : Lustre: 15475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 10:52:06 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:52:06 dc2oss08 kernel: : Lustre: Skipped 43 previous similar messages Mar 5 10:52:06 dc2oss08 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394034726 last 1394034726 Mar 5 10:52:06 dc2oss08 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 43 previous similar messages Mar 5 10:52:07 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394034727 last 1394034727 Mar 5 10:52:07 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 10:51:56 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 10:51:56 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 10:52:44 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:52:44 dc2oss03 kernel: : Lustre: Skipped 137 previous similar messages Mar 5 10:52:44 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394034764 last 1394034764 Mar 5 10:52:44 dc2oss03 kernel: : Lustre: 16610:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 136 previous similar messages Mar 5 10:52:31 dc2oss06 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f4296000 cur 1394034751 last 1394034751 Mar 5 10:52:31 dc2oss06 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 85 previous similar messages Mar 5 10:52:21 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0d:0x5:0x0]] [valid 68719476736] Mar 5 10:52:21 dc2mds01 kernel: : Lustre: 11133:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 82 previous similar messages Mar 5 10:52:43 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 332576, left: 332576, last alive is 36 secs ago Mar 5 10:52:43 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 332576, left: 332576, last alive is 36 secs ago Mar 5 10:52:59 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394034779 last 1394034779 Mar 5 10:52:59 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 10:53:02 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 10:53:02 dc2oss08 kernel: : LustreError: Skipped 12 previous similar messages Mar 5 10:53:19 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394034799 last 1394034799 Mar 5 10:53:19 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 10:53:33 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:53:33 dc2oss16 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 10:53:23 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:53:23 dc2oss04 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 10:53:41 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:53:41 dc2oss02 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 10:53:41 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394034821 last 1394034821 Mar 5 10:53:41 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 10:53:49 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394034829 last 1394034829 Mar 5 10:53:49 dc2oss05 kernel: : Lustre: 16118:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 10:54:32 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:54:32 dc2oss11 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 10:54:32 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 10:54:32 dc2oss11 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:54:42 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 10:54:42 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:54:59 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:54:59 dc2oss13 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 10:54:54 dc2oss11 kernel: : Lustre: Service thread pid 25345 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:54:54 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:54:54 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394034894.25345 Mar 5 10:55:00 dc2oss11 kernel: : Lustre: Service thread pid 25375 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:55:00 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:55:00 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394034900.25375 Mar 5 10:55:04 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:55:04 dc2oss14 kernel: : Lustre: Skipped 82 previous similar messages Mar 5 10:54:56 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:54:56 dc2oss01 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 10:54:56 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394034896 last 1394034896 Mar 5 10:54:56 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 10:55:20 dc2oss11 kernel: : Lustre: Service thread pid 25419 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 10:55:20 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 10:55:20 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394034920.25419 Mar 5 10:55:29 dc2oss09 kernel: : Lustre: dc2-OST005d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:55:29 dc2oss09 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 10:55:33 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 10:55:33 dc2oss04 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:55:56 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0x93:0x0]] [valid 68719476736] Mar 5 10:55:58 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:55:58 dc2oss15 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 10:56:30 dc2oss11 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 112s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST006f_UUID lock: ffff8810e28ce240/0x309c239b3adfe860 lrc: 3/0,0 mode: PR/PR res: 20791927/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c579a5297 expref: 303 pid: 16184 timeout 9050223083 Mar 5 10:56:28 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 10:56:28 dc2oss07 kernel: : LustreError: Skipped 8 previous similar messages Mar 5 10:56:59 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:56:59 dc2oss06 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 10:57:34 dc2oss05 kernel: : Lustre: dc2-OST0031: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:57:34 dc2oss05 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 10:57:43 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 10:57:43 dc2oss12 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:57:49 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 10:57:49 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 10:57:57 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:57:57 dc2oss12 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 10:57:55 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:57:55 dc2oss07 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 10:57:55 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394035075 last 1394035075 Mar 5 10:57:55 dc2oss07 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 10:58:15 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 10:58:15 dc2oss09 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 10:58:32 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 10:58:32 dc2oss05 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 10:58:42 dc2oss10 kernel: : Lustre: dc2-OST0067: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 10:58:42 dc2oss10 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 10:59:01 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 10:59:01 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 10:59:19 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 10:59:19 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 10:59:13 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 10:59:13 dc2oss13 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 10:59:48 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 10:59:48 dc2oss01 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:00:24 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881829c07800 cur 1394035224 last 1394035224 Mar 5 11:00:24 dc2oss09 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 11:00:31 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394035231 last 1394035231 Mar 5 11:00:31 dc2oss12 kernel: : Lustre: 16052:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 11:00:31 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0088_UUID' is not available for connect (no target) Mar 5 11:00:31 dc2oss14 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 11:00:42 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394035242 last 1394035242 Mar 5 11:00:42 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 11:00:54 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 11:00:54 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:00:48 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 11:00:48 dc2mds01 kernel: : Lustre: Skipped 20 previous similar messages Mar 5 11:00:48 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394035248 last 1394035247 Mar 5 11:00:48 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 20 previous similar messages Mar 5 11:01:20 dc2oss04 kernel: : Lustre: 16419:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394035280 last 1394035280 Mar 5 11:01:20 dc2oss04 kernel: : Lustre: 16419:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 11:01:43 149.165.238.1 kernel: kernel: [4309675.857040] LustreError: 11-0: an error occurred while communicating with 10.10.0.171@o2ib. The ldlm_enqueue operation failed with -107 Mar 5 11:01:43 149.165.238.1 kernel: kernel: [4309675.857049] Lustre: dc2-MDT0000-mdc-ffff880956851000: Connection to service dc2-MDT0000 via nid 10.10.0.171@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:01:43 149.165.238.1 kernel: kernel: [4309675.857454] LustreError: 167-0: This client was evicted by dc2-MDT0000; in progress operations using this service will fail. Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056057] Lustre: Server dc2-MDT0000_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056481] LustreError: 13101:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -4 from cancel RPC: canceling anyway Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056486] LustreError: 19741:0:(file.c:116:ll_close_inode_openhandle()) inode 144775648852901540 mdc close failed: rc = -4 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056498] LustreError: 22016:0:(mdc_locks.c:665:mdc_enqueue()) ldlm_cli_enqueue error: -4 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056508] LustreError: 13101:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -4 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056609] LustreError: 22604:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace dc2-MDT0000-mdc-ffff880956851000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056615] LustreError: 22604:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff880d7ba66840 (8616582198/82890/0/0) (rc: 1) Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.056716] LustreError: 22016:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8810ebb85000 x1461669105194125/t0 o101->dc2-MDT0000_UUID@10.10.0.171@o2ib:12/10 lens 560/4880 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.063735] LustreError: 22604:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace dc2-MDT0000-mdc-ffff880956851000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.063739] LustreError: 22604:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Skipped 138 previous similar messages Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064163] LustreError: 22604:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff8801873d4200 (8589958371/9125/0/0) (rc: 1) Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064169] LustreError: 22604:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Skipped 138 previous similar messages Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064324] LustreError: 22604:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8806171fd000 x1461669105194152/t0 o35->dc2-MDT0000_UUID@10.10.0.171@o2ib:23/10 lens 360/9864 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064331] LustreError: 22604:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 25 previous similar messages Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064338] LustreError: 22604:0:(file.c:116:ll_close_inode_openhandle()) inode 144929734042656074 mdc close failed: rc = -108 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.064344] LustreError: 22604:0:(file.c:116:ll_close_inode_openhandle()) Skipped 1 previous similar message Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.116552] LustreError: 19741:0:(mdc_locks.c:665:mdc_enqueue()) ldlm_cli_enqueue error: -108 Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.116556] LustreError: 19741:0:(mdc_locks.c:665:mdc_enqueue()) Skipped 25 previous similar messages Mar 5 11:01:44 149.165.238.1 kernel: kernel: [4309676.326105] Lustre: dc2-MDT0000-mdc-ffff880956851000: Connection restored to service dc2-MDT0000 using nid 10.10.0.171@o2ib. Mar 5 11:01:35 dc2oss15 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394035295 last 1394035295 Mar 5 11:01:35 dc2oss15 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 11:01:49 149.165.238.1 kernel: kernel: [4309681.645357] LustreError: 22596:0:(file.c:116:ll_close_inode_openhandle()) inode 144562260717688049 mdc close failed: rc = -116 Mar 5 11:01:49 149.165.238.1 kernel: kernel: [4309681.645362] LustreError: 22596:0:(file.c:116:ll_close_inode_openhandle()) Skipped 63 previous similar messages Mar 5 11:01:31 dc2mds01 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 186s: evicting client at 149.165.238.1@tcp ns: mdt-ffff881837d4e000 lock: ffff880b781b6d80/0x78c70f36542006e lrc: 3/0,0 mode: PR/PR res: 8631910406/50900 bits 0x3 rrc: 6 type: IBT flags: 0x4000020 remote: 0xe213f1feb82f7113 expref: 36091 pid: 1368 timeout: 4372651881 Mar 5 11:01:32 dc2mds01 kernel: : LustreError: 1368:0:(mdt_handler.c:2817:mdt_recovery()) operation 101 on unconnected MDS from 12345-149.165.238.1@tcp Mar 5 11:01:42 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394035302 last 1394035302 Mar 5 11:01:42 dc2oss14 kernel: : Lustre: 15511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:01:52 149.165.238.1 kernel: kernel: [4309684.798559] LustreError: 22590:0:(file.c:116:ll_close_inode_openhandle()) inode 144562260717691188 mdc close failed: rc = -116 Mar 5 11:01:52 149.165.238.1 kernel: kernel: [4309684.798563] LustreError: 22590:0:(file.c:116:ll_close_inode_openhandle()) Skipped 1 previous similar message Mar 5 11:01:45 dc2oss08 kernel: : Lustre: 29537:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394035297/real 0] req@ffff8804f9c43000 x1456766190337555/t0(0) o105->dc2-OST004e@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394035305 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 11:01:45 dc2oss08 kernel: : LustreError: 138-a: dc2-OST004e: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 11:02:07 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 11:02:07 dc2oss03 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:02:01 149.165.238.1 kernel: kernel: [4309693.552942] LustreError: 22599:0:(file.c:116:ll_close_inode_openhandle()) inode 144562260717692835 mdc close failed: rc = -116 Mar 5 11:02:01 149.165.238.1 kernel: kernel: [4309693.552946] LustreError: 22599:0:(file.c:116:ll_close_inode_openhandle()) Skipped 1 previous similar message Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275106] Lustre: MDC_mds03.local_mds-wan_MNT_client-ffff8803b1d25800: Connection to service mds-wan via nid 149.165.235.173@tcp was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275566] LustreError: 167-0: This client was evicted by mds-wan; in progress operations using this service will fail. Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275680] LustreError: 23767:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace MDC_mds03.local_mds-wan_MNT_client-ffff8803b1d25800 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275685] LustreError: 23767:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Skipped 3772 previous similar messages Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275689] LustreError: 23767:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff8804869a9c80 (139734018/3133879307/0/0) (rc: 1) Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275692] LustreError: 23767:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Skipped 3772 previous similar messages Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275834] LustreError: 23767:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8805c35bb400 x1461669105197229/t0 o35->mds-wan_UUID@mds03.local_UUID:23/10 lens 408/3592 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0 Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275837] LustreError: 23767:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 65 previous similar messages Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275842] LustreError: 23767:0:(file.c:116:ll_close_inode_openhandle()) inode 135385472 mdc close failed: rc = -108 Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.275844] LustreError: 23767:0:(file.c:116:ll_close_inode_openhandle()) Skipped 1 previous similar message Mar 5 11:02:12 149.165.238.1 kernel: kernel: [4309704.276641] Lustre: MDC_mds03.local_mds-wan_MNT_client-ffff8803b1d25800: Connection restored to service mds-wan using nid 149.165.235.173@tcp. Mar 5 11:02:16 149.165.238.1 kernel: kernel: [4309708.676028] Lustre: 27471:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105196821 sent from dc2-OST002c-osc-ffff880956851000 to NID 10.10.0.5@o2ib 9s ago has timed out (9s prior to deadline). Mar 5 11:02:16 149.165.238.1 kernel: kernel: [4309708.676043] Lustre: dc2-OST002c-osc-ffff880956851000: Connection to service dc2-OST002c via nid 10.10.0.5@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:02:16 149.165.238.1 kernel: kernel: [4309708.676383] Lustre: dc2-OST002c-osc-ffff880956851000: Connection restored to service dc2-OST002c using nid 10.10.0.5@o2ib. Mar 5 11:02:16 149.165.238.1 kernel: kernel: [4309708.676393] Lustre: Server dc2-OST002c_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:02:10 dc2oss08 kernel: : Lustre: dc2-OST0051: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:02:10 dc2oss08 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 11:02:10 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394035330 last 1394035330 Mar 5 11:02:10 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 11:02:29 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394035349 last 1394035349 Mar 5 11:02:29 dc2oss16 kernel: : Lustre: 17215:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 11:02:34 dc2oss06 kernel: : Lustre: 15837:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394035354 last 1394035354 Mar 5 11:02:34 dc2oss06 kernel: : Lustre: 15837:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 11:02:50 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:02:50 dc2oss03 kernel: : Lustre: Skipped 63 previous similar messages Mar 5 11:02:50 dc2oss03 kernel: : Lustre: 16936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394035370 last 1394035370 Mar 5 11:02:50 dc2oss03 kernel: : Lustre: 16936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 64 previous similar messages Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.432884] Lustre: dc2-OST0000-osc-ffff880956851000: Connection to service dc2-OST0000 via nid 10.10.0.1@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433181] LustreError: 167-0: This client was evicted by dc2-OST0000; in progress operations using this service will fail. Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433223] Lustre: Server dc2-OST0000_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433255] LustreError: 23727:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -4 from cancel RPC: canceling anyway Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433260] LustreError: 23727:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -4 Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433450] LustreError: 23887:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace dc2-OST0000-osc-ffff880956851000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433455] LustreError: 23887:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Skipped 49 previous similar messages Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433459] LustreError: 23887:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff88031a1a06c0 (18965876/0/0/0) (rc: 1) Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.433461] LustreError: 23887:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Skipped 49 previous similar messages Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.658582] Lustre: dc2-OST0000-osc-ffff880956851000: Connection restored to service dc2-OST0000 using nid 10.10.0.1@o2ib. Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.658894] LustreError: 21247:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5 Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.659135] LustreError: 21247:0:(file.c:116:ll_close_inode_openhandle()) inode 144943094477919681 mdc close failed: rc = -116 Mar 5 11:02:42 149.165.238.1 kernel: kernel: [4309734.659138] LustreError: 21247:0:(file.c:116:ll_close_inode_openhandle()) Skipped 5 previous similar messages Mar 5 11:02:30 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:02:30 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 11 previous similar messages Mar 5 11:02:33 dc2oss01 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 149.165.238.1@tcp ns: filter-dc2-OST0000_UUID lock: ffff8814f9cd6b40/0x7b783ece9bb5ea1 lrc: 3/0,0 mode: PR/PR res: 20762492/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xe213f1feb84517ca expref: 165 pid: 15387 timeout 4377882561 Mar 5 11:02:33 dc2oss01 kernel: : LustreError: 16770:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.238.1@tcp arrived at 1394035353 with bad export cookie 556058132555803534 Mar 5 11:03:00 dc2oss10 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394035380 last 1394035380 Mar 5 11:03:00 dc2oss10 kernel: : Lustre: 16277:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 11:03:01 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 412512, left: 412512, last alive is 53 secs ago Mar 5 11:03:01 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 412512, left: 412512, last alive is 53 secs ago Mar 5 11:03:29 149.165.238.1 kernel: kernel: [4309780.977198] Lustre: dc2-OST004f-osc-ffff880956851000: Connection to service dc2-OST004f via nid 10.10.0.8@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:03:19 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394035399 last 1394035399 Mar 5 11:03:19 dc2oss11 kernel: : Lustre: 14503:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 11:03:30 149.165.238.1 kernel: kernel: [4309782.054788] LustreError: 167-0: This client was evicted by dc2-OST004f; in progress operations using this service will fail. Mar 5 11:03:30 149.165.238.1 kernel: kernel: [4309782.054828] Lustre: Server dc2-OST004f_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:03:30 149.165.238.1 kernel: kernel: [4309782.054872] LustreError: 23727:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -4 from cancel RPC: canceling anyway Mar 5 11:03:30 149.165.238.1 kernel: kernel: [4309782.054877] LustreError: 23727:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -4 Mar 5 11:03:30 149.165.238.1 kernel: kernel: [4309782.168664] Lustre: dc2-OST004f-osc-ffff880956851000: Connection restored to service dc2-OST004f using nid 10.10.0.8@o2ib. Mar 5 11:03:32 149.165.238.1 kernel: kernel: [4309784.236029] Lustre: 14346:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105207718 sent from dc2-OST0016-osc-ffff880956851000 to NID 10.10.0.3@o2ib 7s ago has timed out (7s prior to deadline). Mar 5 11:03:32 149.165.238.1 kernel: kernel: [4309784.236367] Lustre: Server dc2-OST0016_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:03:27 dc2oss08 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 1438s: evicting client at 149.165.238.1@tcp ns: filter-dc2-OST004f_UUID lock: ffff8811406d5b40/0x9b008705060058b lrc: 3/0,0 mode: PW/PW res: 22177191/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 remote: 0xe213f1feb7e57eea expref: 183 pid: 15979 timeout 9050642961 Mar 5 11:03:28 dc2oss08 kernel: : LustreError: 29537:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.238.1@tcp arrived at 1394035408 with bad export cookie 698067220716041611 Mar 5 11:03:28 dc2oss08 kernel: : LustreError: 29537:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.238.1@tcp arrived at 1394035408 with bad export cookie 698067220716041611 Mar 5 11:03:27 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 421472, left: 421472, last alive is 3 secs ago Mar 5 11:03:27 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 421472, left: 421472, last alive is 3 secs ago Mar 5 11:03:41 dc2oss02 kernel: : Lustre: dc2-OST000d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:03:41 dc2oss02 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 11:03:41 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818273a1000 cur 1394035421 last 1394035421 Mar 5 11:03:41 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 68 previous similar messages Mar 5 11:03:27 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:03:27 dc2oss04 kernel: : Lustre: Skipped 133 previous similar messages Mar 5 11:03:26 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f796:0x5f56:0x0]] [valid 68719476736] Mar 5 11:03:26 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 66 previous similar messages Mar 5 11:03:45 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:03:45 dc2oss16 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 11:03:53 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394035433 last 1394035433 Mar 5 11:03:53 dc2oss05 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 11:04:35 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:04:35 dc2oss11 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 11:04:41 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0079_UUID' is not available for connect (no target) Mar 5 11:04:41 dc2oss11 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 11:04:48 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 11:04:48 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 11:04:59 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:04:59 dc2oss13 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 11:05:08 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:05:08 dc2oss14 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 11:05:14 dc2oss01 kernel: : Lustre: dc2-OST0007: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:05:14 dc2oss01 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 11:05:14 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c98c00 cur 1394035514 last 1394035514 Mar 5 11:05:14 dc2oss01 kernel: : Lustre: 15429:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 11:05:25 dc2oss11 kernel: : Lustre: 14718:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-231), not sending early reply Mar 5 11:05:30 dc2oss11 kernel: : LustreError: 25345:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 836+0s req@ffff880bd5ac9c00 x1442912236918122/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394035530 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:05:30 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:05:30 dc2oss11 kernel: : Lustre: Skipped 5 previous similar messages Mar 5 11:05:30 dc2oss11 kernel: : Lustre: Service thread pid 25345 completed after 836.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 11:05:31 dc2oss11 kernel: : Lustre: 25353:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-231), not sending early reply Mar 5 11:05:30 dc2oss09 kernel: : Lustre: dc2-OST0059: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:05:30 dc2oss09 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 11:05:36 dc2oss11 kernel: : LustreError: 25375:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 836+0s req@ffff880bd6930400 x1442912236918350/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394035536 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:05:36 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:05:36 dc2oss11 kernel: : Lustre: Service thread pid 25375 completed after 836.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 11:05:42 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 11:05:42 dc2oss08 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 11:05:51 dc2oss11 kernel: : Lustre: 8734:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-231), not sending early reply Mar 5 11:05:56 dc2oss11 kernel: : LustreError: 25419:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 836+0s req@ffff880bd60e4400 x1442912236918369/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394035556 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:05:56 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:05:56 dc2oss11 kernel: : Lustre: Service thread pid 25419 completed after 836.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 11:06:04 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:06:04 dc2oss15 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028034] Lustre: 28556:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105221847 sent from dc2-OST004e-osc-ffff880956851000 to NID 10.10.0.8@o2ib 13s ago has timed out (13s prior to deadline). Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028049] Lustre: dc2-OST004e-osc-ffff880956851000: Connection to service dc2-OST004e via nid 10.10.0.8@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028053] Lustre: Skipped 1 previous similar message Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028445] Lustre: dc2-OST004e-osc-ffff880956851000: Connection restored to service dc2-OST004e using nid 10.10.0.8@o2ib. Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028449] Lustre: Skipped 1 previous similar message Mar 5 11:06:31 149.165.238.1 kernel: kernel: [4309963.028462] Lustre: Server dc2-OST004e_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:06:42 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0018_UUID' is not available for connect (no target) Mar 5 11:06:42 dc2oss04 kernel: : LustreError: Skipped 9 previous similar messages Mar 5 11:06:59 dc2oss06 kernel: : Lustre: dc2-OST0035: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:06:59 dc2oss06 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 11:07:04 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0050_UUID' is not available for connect (no target) Mar 5 11:07:04 dc2oss07 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 11:07:34 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 11:07:34 dc2oss11 kernel: : LustreError: 25375:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd33efc00 x1442912236929226/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394036407 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:07:34 dc2oss11 kernel: : LustreError: 25375:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:07:34 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:07:35 dc2oss11 kernel: : LustreError: 25455:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8817fa8f3850 x1442912236925624/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394035869 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:07:35 dc2xfer02 kernel: : Lustre: 2703:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:07:35 dc2xfer02 kernel: : Lustre: 2703:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:07:37 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:07:37 dc2oss05 kernel: : Lustre: Skipped 77 previous similar messages Mar 5 11:07:58 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:07:58 dc2oss12 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 11:07:58 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:07:58 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 11:07:58 dc2oss07 kernel: : Lustre: 14421:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394035678 last 1394035678 Mar 5 11:07:58 dc2oss07 kernel: : Lustre: 14421:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 11:08:04 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 11:08:04 dc2oss15 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:08:18 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 11:08:18 dc2oss12 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 11:08:47 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:08:47 dc2oss10 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 11:08:57 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 11:08:57 dc2oss05 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:09:22 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 11:09:22 dc2oss02 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:09:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 11:09:33 dc2oss13 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 11:09:41 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0063_UUID' is not available for connect (no target) Mar 5 11:09:41 dc2oss09 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 11:10:06 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 11:10:06 dc2oss06 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 11:10:14 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 11:10:14 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 11:10:25 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394035825 last 1394035825 Mar 5 11:10:25 dc2oss09 kernel: : Lustre: 14527:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 11:10:38 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394035838 last 1394035838 Mar 5 11:10:38 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 11:10:46 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394035846 last 1394035846 Mar 5 11:10:46 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 11:10:56 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 11:10:56 dc2oss14 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 11:10:49 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 11:10:49 dc2mds01 kernel: : Lustre: Skipped 36 previous similar messages Mar 5 11:10:49 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394035849 last 1394035849 Mar 5 11:10:49 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 37 previous similar messages Mar 5 11:11:23 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0148b000 cur 1394035883 last 1394035883 Mar 5 11:11:23 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 11:11:40 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881818a2b800 cur 1394035900 last 1394035900 Mar 5 11:11:40 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 11:11:43 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394035903 last 1394035903 Mar 5 11:11:43 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:11:47 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 11:11:47 dc2oss16 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 11:12:13 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0025_UUID' is not available for connect (no target) Mar 5 11:12:13 dc2oss03 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:12:18 149.165.238.1 kernel: kernel: [4310309.964474] Lustre: 19741:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105271908 sent from dc2-OST0092-osc-ffff880956851000 to NID 10.10.0.14@o2ib 9s ago has timed out (9s prior to deadline). Mar 5 11:12:18 149.165.238.1 kernel: kernel: [4310309.964490] Lustre: dc2-OST0092-osc-ffff880956851000: Connection to service dc2-OST0092 via nid 10.10.0.14@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 11:12:18 149.165.238.1 kernel: kernel: [4310309.964944] Lustre: dc2-OST0092-osc-ffff880956851000: Connection restored to service dc2-OST0092 using nid 10.10.0.14@o2ib. Mar 5 11:12:18 149.165.238.1 kernel: kernel: [4310309.964965] Lustre: Server dc2-OST0092_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 11:12:17 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:12:17 dc2oss08 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 11:12:17 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394035937 last 1394035937 Mar 5 11:12:17 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 11:12:35 dc2oss06 kernel: : Lustre: 16092:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394035955 last 1394035955 Mar 5 11:12:35 dc2oss06 kernel: : Lustre: 16092:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:12:30 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a6: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818268aac00 cur 1394035950 last 1394035950 Mar 5 11:12:30 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:12:51 dc2oss03 kernel: : Lustre: dc2-OST0016: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:12:51 dc2oss03 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 11:12:51 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394035971 last 1394035971 Mar 5 11:12:51 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 11:13:08 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 234368, left: 234368, last alive is 10 secs ago Mar 5 11:13:08 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 234368, left: 234368, last alive is 10 secs ago Mar 5 11:13:24 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394036004 last 1394036004 Mar 5 11:13:24 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:13:32 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:13:32 dc2oss04 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 11:13:52 dc2oss02 kernel: : Lustre: dc2-OST0011: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:13:52 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:13:52 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 11:13:52 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 11:13:52 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf43b6400 cur 1394036032 last 1394036032 Mar 5 11:13:52 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 11:13:45 dc2oss10 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394036025 last 1394036025 Mar 5 11:13:45 dc2oss10 kernel: : Lustre: 16386:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 11:13:47 dc2oss16 kernel: : Lustre: dc2-OST009e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:13:47 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 11:13:43 dc2mds01 kernel: : Lustre: 11179:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0f:0x213:0x0]] [valid 68719476736] Mar 5 11:13:43 dc2mds01 kernel: : Lustre: 11179:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 49 previous similar messages Mar 5 11:13:55 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828c79c00 cur 1394036035 last 1394036035 Mar 5 11:13:55 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394036035 last 1394036035 Mar 5 11:13:55 dc2oss05 kernel: : Lustre: 15795:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 73 previous similar messages Mar 5 11:14:23 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:14:23 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 21 previous similar messages Mar 5 11:14:52 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 11:14:52 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 11:15:09 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:15:09 dc2oss14 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 11:15:04 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 11:15:04 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 11:15:06 dc2oss11 kernel: : Lustre: dc2-OST0072: Client c1ce5d4a-1b3a-c884-4cf4-e623e681091b (at 149.165.235.51@tcp) reconnecting Mar 5 11:15:06 dc2oss11 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 11:15:06 dc2oss13 kernel: : Lustre: dc2-OST007e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:15:06 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 11:15:24 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:15:24 dc2oss01 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 11:15:24 dc2oss01 kernel: : Lustre: 15540:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394036124 last 1394036124 Mar 5 11:15:24 dc2oss01 kernel: : Lustre: 15540:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 11:15:39 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:15:39 dc2oss09 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 11:16:06 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:16:06 dc2oss15 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 11:16:22 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 11:16:22 dc2oss08 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 11:17:13 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 11:17:13 dc2oss04 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:17:33 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:17:33 dc2oss06 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 11:17:58 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:17:58 dc2oss12 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 11:17:57 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:17:57 dc2oss05 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 11:18:04 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:18:04 dc2oss07 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 11:18:04 dc2oss07 kernel: : Lustre: 15881:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394036284 last 1394036284 Mar 5 11:18:04 dc2oss07 kernel: : Lustre: 15881:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 11:18:26 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006a_UUID' is not available for connect (no target) Mar 5 11:18:26 dc2oss12 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:18:36 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 11:18:36 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 11:19:09 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 11:19:09 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:19:10 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:19:10 dc2oss10 kernel: : Lustre: Skipped 71 previous similar messages Mar 5 11:19:22 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004e_UUID' is not available for connect (no target) Mar 5 11:19:22 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:19:42 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0000_UUID' is not available for connect (no target) Mar 5 11:19:42 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 11:19:46 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 11:19:46 dc2oss13 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:19:58 dc2oss05 kernel: : Lustre: dc2-OST0030: Client 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 11:19:58 dc2oss05 kernel: : LustreError: 25201:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880391fb8850 x1461680408660327/t0(0) o3->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/400 e 1 to 0 dl 1394036426 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:19:58 dc2oss05 kernel: : LustreError: 25201:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 11:19:58 dc2oss05 kernel: : Lustre: dc2-OST0030: Bulk IO read error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 11:19:58 dc2oss05 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 11:19:59 dc2oss05 kernel: : LustreError: 29969:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806da06f800 x1461680408660340/t0(0) o3->5d2fd56f-464f-0804-4a12-da6fb48f57bd@149.165.229.53@tcp:0/0 lens 448/400 e 1 to 0 dl 1394036428 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:19:59 dc2oss05 kernel: : LustreError: 29969:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:19:59 dc2oss05 kernel: : Lustre: dc2-OST0030: Bulk IO read error with 5d2fd56f-464f-0804-4a12-da6fb48f57bd (at 149.165.229.53@tcp), client will retry: rc -110 Mar 5 11:19:59 dc2oss05 kernel: : Lustre: Skipped 1 previous similar message Mar 5 11:20:23 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000e_UUID' is not available for connect (no target) Mar 5 11:20:23 dc2oss01 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 11:20:33 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c00a7e400 cur 1394036433 last 1394036433 Mar 5 11:20:33 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 11:20:36 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002c_UUID' is not available for connect (no target) Mar 5 11:20:36 dc2oss06 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 11:20:40 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394036440 last 1394036440 Mar 5 11:20:40 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 11:20:55 dc2oss13 kernel: : Lustre: 14428:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394036455 last 1394036455 Mar 5 11:20:55 dc2oss13 kernel: : Lustre: 14428:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 11:20:54 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:20:54 dc2mds01 kernel: : Lustre: Skipped 18 previous similar messages Mar 5 11:20:54 dc2mds01 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394036454 last 1394036454 Mar 5 11:20:54 dc2mds01 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 18 previous similar messages Mar 5 11:21:20 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0084_UUID' is not available for connect (no target) Mar 5 11:21:20 dc2oss14 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 11:21:29 dc2oss09 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 111s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST005c_UUID lock: ffff880f38382b40/0xacae7902c57af588 lrc: 3/0,0 mode: PR/PR res: 23607308/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c57ecb504 expref: 378 pid: 16453 timeout 9051713108 Mar 5 11:21:32 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST005f_UUID' is not available for connect (no target) Mar 5 11:21:32 dc2oss09 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:21:41 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0x9b:0x0]] [valid 68719476736] Mar 5 11:21:28 dc2oss04 kernel: : Lustre: 16451:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394036488 last 1394036488 Mar 5 11:21:28 dc2oss04 kernel: : Lustre: 16451:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 11:21:24 dc2mds01 kernel: : Lustre: 14557:0:(mdd_device.c:900:obf_lookup()) bad FID format [[0x2021bc72b:0x0:0x0]], should be [0x1:0x2:0x0] Mar 5 11:21:24 dc2mds01 kernel: : Lustre: 14557:0:(mdd_device.c:900:obf_lookup()) Skipped 193 previous similar messages Mar 5 11:21:40 dc2oss15 kernel: : Lustre: 16348:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394036500 last 1394036500 Mar 5 11:21:40 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394036500 last 1394036500 Mar 5 11:21:40 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 11:21:44 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394036504 last 1394036504 Mar 5 11:21:44 dc2oss14 kernel: : Lustre: 15585:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 11:21:47 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 11:21:47 dc2oss16 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:22:36 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 11:22:36 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:22:24 dc2oss08 kernel: : Lustre: dc2-OST0052: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:22:24 dc2oss08 kernel: : Lustre: Skipped 132 previous similar messages Mar 5 11:22:24 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04bdd000 cur 1394036544 last 1394036544 Mar 5 11:22:24 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 11:22:38 dc2oss06 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394036558 last 1394036557 Mar 5 11:22:38 dc2oss06 kernel: : Lustre: 16024:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 11:22:43 dc2oss16 kernel: : Lustre: 15767:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394036563 last 1394036563 Mar 5 11:22:43 dc2oss16 kernel: : Lustre: 15767:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 11:23:14 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:23:14 dc2oss03 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 11:23:14 dc2oss03 kernel: : Lustre: 17105:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394036594 last 1394036594 Mar 5 11:23:14 dc2oss03 kernel: : Lustre: 17105:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 11:23:30 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394036610 last 1394036610 Mar 5 11:23:30 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 137 previous similar messages Mar 5 11:23:54 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:23:54 dc2oss02 kernel: : Lustre: Skipped 163 previous similar messages Mar 5 11:23:54 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394036634 last 1394036634 Mar 5 11:23:54 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 162 previous similar messages Mar 5 11:23:45 dc2oss10 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1df3c00 cur 1394036625 last 1394036625 Mar 5 11:23:45 dc2oss10 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 11:23:58 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:23:58 dc2oss16 kernel: : Lustre: Skipped 132 previous similar messages Mar 5 11:23:53 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:23:53 dc2oss04 kernel: : Lustre: Skipped 125 previous similar messages Mar 5 11:24:10 dc2oss05 kernel: : Lustre: 15884:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394036650 last 1394036650 Mar 5 11:24:10 dc2oss05 kernel: : Lustre: 15884:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 130 previous similar messages Mar 5 11:24:35 dc2oss11 kernel: : Lustre: 8722:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:24:37 dc2oss11 kernel: : Lustre: 8805:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:24:40 dc2oss11 kernel: : LustreError: 25438:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880152813000 x1442912236931920/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394036680 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:24:40 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:24:40 dc2oss11 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 11:24:41 dc2oss11 kernel: : Lustre: 8635:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:24:42 dc2oss11 kernel: : LustreError: 14725:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880f9d7b7050 x1442912236931924/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394036682 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:24:42 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:24:45 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 11:24:45 dc2oss11 kernel: : LustreError: 8705:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc71a6400 x1442912236939733/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037435 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:24:45 dc2oss11 kernel: : LustreError: 8705:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 11:24:45 dc2oss11 kernel: : LustreError: 8746:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd2c97000 x1442912236939730/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037432 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:24:46 dc2oss11 kernel: : LustreError: 8635:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880783754000 x1442912236939735/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037439 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:24:46 dc2oss11 kernel: : LustreError: 8635:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 11:24:28 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:24:28 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 17 previous similar messages Mar 5 11:24:46 dc2xfer02 kernel: : Lustre: 2701:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:46 dc2xfer02 kernel: : Lustre: 2701:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:47 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:47 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:51 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:51 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:54 dc2xfer03 kernel: : Lustre: 2648:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:24:54 dc2xfer03 kernel: : Lustre: 2648:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:25:00 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 11:25:00 dc2oss11 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:25:10 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:25:10 dc2oss13 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 11:25:12 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 11:25:12 dc2oss10 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 11:25:12 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:25:12 dc2oss11 kernel: : Lustre: Skipped 150 previous similar messages Mar 5 11:25:13 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:25:13 dc2oss14 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 11:25:09 dc2mds01 kernel: : Lustre: 11160:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0f:0x461:0x0]] [valid 68719476736] Mar 5 11:25:09 dc2mds01 kernel: : Lustre: 11160:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 41 previous similar messages Mar 5 11:25:29 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:25:29 dc2oss01 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 11:25:29 dc2oss01 kernel: : Lustre: 15351:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394036729 last 1394036729 Mar 5 11:25:29 dc2oss01 kernel: : Lustre: 15351:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:25:39 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:25:39 dc2oss09 kernel: : Lustre: Skipped 149 previous similar messages Mar 5 11:26:15 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:26:15 dc2oss15 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 11:26:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 122112, left: 122112, last alive is 0 secs ago Mar 5 11:26:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 122112, left: 122112, last alive is 0 secs ago Mar 5 11:26:54 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 11:26:54 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:27:13 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 11:27:13 dc2oss04 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 11:27:34 dc2oss06 kernel: : Lustre: dc2-OST003e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:27:34 dc2oss06 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 11:27:59 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:27:59 dc2oss12 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 11:27:59 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:27:59 dc2oss05 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 11:28:05 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:28:05 dc2oss07 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 11:28:05 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394036885 last 1394036885 Mar 5 11:28:05 dc2oss07 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 11:28:36 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 11:28:36 dc2oss15 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:29:17 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0038_UUID' is not available for connect (no target) Mar 5 11:29:17 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:29:17 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 11:29:17 dc2oss12 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 11:29:37 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004e_UUID' is not available for connect (no target) Mar 5 11:29:37 dc2oss07 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 11:29:58 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 11:29:58 dc2oss02 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 11:30:02 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:30:02 dc2oss10 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 11:30:23 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000d_UUID' is not available for connect (no target) Mar 5 11:30:23 dc2oss01 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:30:35 dc2oss09 kernel: : Lustre: 14507:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394037035 last 1394037035 Mar 5 11:30:35 dc2oss09 kernel: : Lustre: 14507:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 11:30:43 dc2oss12 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394037043 last 1394037043 Mar 5 11:30:43 dc2oss12 kernel: : Lustre: 15986:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 11:31:06 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394037066 last 1394037066 Mar 5 11:31:06 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 11:30:58 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:30:58 dc2mds01 kernel: : Lustre: Skipped 35 previous similar messages Mar 5 11:30:58 dc2mds01 kernel: : Lustre: 1367:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394037058 last 1394037058 Mar 5 11:30:58 dc2mds01 kernel: : Lustre: 1367:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 35 previous similar messages Mar 5 11:31:22 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0089_UUID' is not available for connect (no target) Mar 5 11:31:22 dc2oss13 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 11:31:23 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 11:31:23 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:31:26 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 11:31:26 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:31:35 dc2oss04 kernel: : Lustre: 16434:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394037095 last 1394037095 Mar 5 11:31:35 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394037095 last 1394037095 Mar 5 11:31:35 dc2oss04 kernel: : Lustre: 16760:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:31:44 dc2oss15 kernel: : Lustre: 15948:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0096: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c267d7c00 cur 1394037104 last 1394037104 Mar 5 11:31:44 dc2oss15 kernel: : Lustre: 15948:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 11:31:48 dc2oss14 kernel: : Lustre: 15768:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394037108 last 1394037108 Mar 5 11:31:48 dc2oss14 kernel: : Lustre: 15768:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 11:32:05 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009c_UUID' is not available for connect (no target) Mar 5 11:32:05 dc2oss16 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:32:39 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0024_UUID' is not available for connect (no target) Mar 5 11:32:39 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:32:32 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:32:32 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:32:32 dc2oss08 kernel: : Lustre: Skipped 74 previous similar messages Mar 5 11:32:32 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394037152 last 1394037152 Mar 5 11:32:32 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 75 previous similar messages Mar 5 11:32:41 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 18976, left: 18976, last alive is 27 secs ago Mar 5 11:32:41 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 18976, left: 18976, last alive is 27 secs ago Mar 5 11:32:45 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394037165 last 1394037165 Mar 5 11:32:45 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 11:32:47 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394037167 last 1394037167 Mar 5 11:32:47 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 11:33:15 dc2oss03 kernel: : Lustre: dc2-OST0016: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:33:15 dc2oss03 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 11:33:15 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31136294 exp ffff880c0266f000 cur 1394037195 last 1394037195 Mar 5 11:33:15 dc2oss03 kernel: : Lustre: 16679:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 11:32:53 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 770912, left: 770912, last alive is 31 secs ago Mar 5 11:32:53 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 770912, left: 770912, last alive is 31 secs ago Mar 5 11:33:32 dc2oss11 kernel: : Lustre: 15422:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394037212 last 1394037212 Mar 5 11:33:32 dc2oss11 kernel: : Lustre: 15422:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 11:33:56 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:33:56 dc2oss02 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 11:33:56 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394037236 last 1394037236 Mar 5 11:33:56 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 11:33:44 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 11:33:44 dc2oss09 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 11:33:49 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394037229 last 1394037229 Mar 5 11:33:49 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 56 previous similar messages Mar 5 11:34:01 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:34:01 dc2oss16 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 11:33:59 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:33:59 dc2oss04 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 11:34:16 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be9c22400 cur 1394037256 last 1394037256 Mar 5 11:34:16 dc2oss05 kernel: : Lustre: 16021:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:34:31 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:34:31 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 11:35:06 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 11:35:06 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 11:35:22 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 11:35:22 dc2oss10 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:35:21 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:35:21 dc2oss14 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 11:35:11 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0f:0x13f3:0x0]] [valid 68719476736] Mar 5 11:35:11 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 32 previous similar messages Mar 5 11:35:22 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:35:22 dc2oss11 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 11:35:26 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:35:26 dc2oss13 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 11:35:30 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:35:30 dc2oss01 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 11:35:30 dc2oss01 kernel: : Lustre: 15351:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394037330 last 1394037330 Mar 5 11:35:30 dc2oss01 kernel: : Lustre: 15351:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 11:35:41 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:35:41 dc2oss09 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 11:36:15 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:36:15 dc2oss15 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 11:37:35 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:37:35 dc2oss06 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 11:37:36 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 11:37:36 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:38:00 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:38:00 dc2oss12 kernel: : Lustre: Skipped 88 previous similar messages Mar 5 11:38:00 dc2oss11 kernel: : Lustre: 8768:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:38:02 dc2oss11 kernel: : Lustre: 25413:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:38:05 dc2oss11 kernel: : LustreError: 25382:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff88160249b050 x1442912236940254/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037485 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:38:05 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:38:05 dc2oss11 kernel: : Lustre: Skipped 6 previous similar messages Mar 5 11:38:07 dc2oss11 kernel: : LustreError: 25448:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd51bf000 x1442912236940257/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037487 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:38:07 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:38:08 dc2oss11 kernel: : Lustre: 25375:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:38:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 11:38:09 dc2oss11 kernel: : LustreError: 14680:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880a4f304800 x1442912236947971/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037495 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:38:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:38:09 dc2oss11 kernel: : LustreError: 25348:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff88019fe61400 x1442912236944378/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037862 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:38:09 dc2oss11 kernel: : LustreError: 25348:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:38:09 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd5ac9000 x1442912236944372/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037854 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:38:09 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 11:38:08 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:38:08 dc2oss05 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 11:38:10 dc2oss07 kernel: : Lustre: dc2-OST0047: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:38:10 dc2oss07 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 11:38:10 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1394037490 last 1394037490 Mar 5 11:38:10 dc2oss07 kernel: : Lustre: 15869:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:38:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 699232, left: 699232, last alive is 41 secs ago Mar 5 11:38:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 699232, left: 699232, last alive is 41 secs ago Mar 5 11:38:45 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a0_UUID' is not available for connect (no target) Mar 5 11:38:45 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 11:38:41 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 11:38:41 dc2oss04 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 11:39:22 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 11:39:22 dc2oss05 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:39:29 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 11:39:29 dc2oss12 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:40:02 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 11:40:02 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 11:40:01 dc2oss11 kernel: : LustreError: 8666:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 100+0s req@ffff880bc704c000 x1442912236948418/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394037601 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 11:40:01 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:40:01 dc2oss11 kernel: : Lustre: Skipped 5 previous similar messages Mar 5 11:40:03 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:40:03 dc2oss10 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 11:40:34 dc2oss11 kernel: : Lustre: dc2-OST0070: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:40:34 dc2oss11 kernel: : LustreError: 25369:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880aa82a2c00 x1461680324742935/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 0 to 0 dl 1394038357 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:34 dc2oss11 kernel: : Lustre: dc2-OST0070: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:34 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 11:40:34 dc2oss06 kernel: : Lustre: dc2-OST003c: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:40:35 dc2oss06 kernel: : LustreError: 14523:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806650c2800 x1461680324742922/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037663 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:35 dc2oss06 kernel: : LustreError: 14523:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 11:40:35 dc2oss06 kernel: : Lustre: dc2-OST003c: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:27 dc2oss05 kernel: : Lustre: dc2-OST002f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:40:28 dc2oss05 kernel: : LustreError: 29794:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b5dd36c00 x1461680324742915/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037662 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:28 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:33 dc2oss08 kernel: : Lustre: dc2-OST004d: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 11:40:33 dc2oss08 kernel: : LustreError: 32210:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88023bd0a400 x1461680324742920/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037669 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:33 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:33 dc2oss08 kernel: : LustreError: 27689:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800b51e3c00 x1461680324742943/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037672 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:33 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:36 dc2oss09 kernel: : Lustre: 14492:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394037636 last 1394037636 Mar 5 11:40:36 dc2oss09 kernel: : Lustre: 14492:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 11:40:27 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 11:40:27 dc2oss01 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 11:40:44 dc2oss08 kernel: : Lustre: dc2-OST004e: Client 2d8f53f2-99c1-152d-ff7e-c86c32f12361 (at 149.165.231.13@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:40:45 dc2oss08 kernel: : LustreError: 32275:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880027e21800 x1461677749267693/t0(0) o3->2d8f53f2-99c1-152d-ff7e-c86c32f12361@149.165.231.13@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037683 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:45 dc2oss08 kernel: : LustreError: 32275:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:40:53 dc2oss08 kernel: : Lustre: dc2-OST004b: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 11:40:53 dc2oss08 kernel: : LustreError: 13786:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800a8a2ac00 x1461680324743261/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037676 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:40:53 dc2oss08 kernel: : Lustre: dc2-OST004b: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:40:53 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 11:40:48 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394037648 last 1394037648 Mar 5 11:40:48 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 11:40:59 dc2oss08 kernel: : Lustre: dc2-OST004e: Client 2d8f53f2-99c1-152d-ff7e-c86c32f12361 (at 149.165.231.13@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:41:01 dc2oss08 kernel: : Lustre: dc2-OST004e: Bulk IO read error with 2d8f53f2-99c1-152d-ff7e-c86c32f12361 (at 149.165.231.13@tcp), client will retry: rc -110 Mar 5 11:41:01 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 11:41:01 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 11:41:01 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 11:41:01 dc2oss07 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 11:41:01 dc2oss07 kernel: : LustreError: 25220:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880090ccc000 x1461680324742911/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394037746 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:41:01 dc2oss07 kernel: : LustreError: 25220:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:41:01 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:41:01 dc2oss07 kernel: : Lustre: Skipped 1 previous similar message Mar 5 11:41:14 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0086: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181d02e800 cur 1394037674 last 1394037674 Mar 5 11:41:14 dc2oss13 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 11:41:26 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002f_UUID' is not available for connect (no target) Mar 5 11:41:26 dc2oss06 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 11:41:26 dc2oss16 kernel: : Lustre: dc2-OST009f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:41:27 dc2oss16 kernel: : LustreError: 9256:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880954125c00 x1461680324742924/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 2 to 0 dl 1394037750 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:41:27 dc2oss16 kernel: : Lustre: dc2-OST009f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:41:15 dc2mds01 kernel: : Lustre: MGS: Client aa892c4e-485c-10c8-4108-b3f313608eda (at 149.165.229.58@tcp) reconnecting Mar 5 11:41:15 dc2mds01 kernel: : Lustre: Skipped 28 previous similar messages Mar 5 11:41:15 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aa892c4e-485c-10c8-4108-b3f313608eda@149.165.229.58@tcp t0 exp ffff8817ff786000 cur 1394037675 last 1394037675 Mar 5 11:41:15 dc2mds01 kernel: : Lustre: 18114:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 28 previous similar messages Mar 5 11:41:30 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0081_UUID' is not available for connect (no target) Mar 5 11:41:30 dc2oss14 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 11:41:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 11:41:33 dc2oss13 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 11:41:37 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394037697 last 1394037697 Mar 5 11:41:37 dc2oss04 kernel: : Lustre: 16077:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 11:41:49 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0096: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c267d7c00 cur 1394037709 last 1394037709 Mar 5 11:41:49 dc2oss15 kernel: : Lustre: 15804:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 11:41:53 dc2oss14 kernel: : Lustre: 15815:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394037713 last 1394037713 Mar 5 11:41:53 dc2oss14 kernel: : Lustre: 15815:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 11:42:05 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 11:42:05 dc2oss16 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 11:42:51 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 11:42:51 dc2oss03 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 11:42:35 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 690272, left: 690272, last alive is 6 secs ago Mar 5 11:42:35 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 690272, left: 690272, last alive is 6 secs ago Mar 5 11:42:44 dc2oss08 kernel: : Lustre: dc2-OST004d: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) reconnecting Mar 5 11:42:44 dc2oss08 kernel: : Lustre: Skipped 144 previous similar messages Mar 5 11:42:44 dc2oss08 kernel: : Lustre: dc2-OST004d: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 11:42:45 dc2oss08 kernel: : LustreError: 13777:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b7730ec00 x1461680324744895/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037845 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:42:45 dc2oss08 kernel: : LustreError: 13777:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 11:42:45 dc2oss08 kernel: : Lustre: dc2-OST004d: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:42:44 dc2oss05 kernel: : Lustre: dc2-OST002f: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 11:42:44 dc2oss05 kernel: : LustreError: 14752:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880325ff1400 x1461680324745002/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037839 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:42:44 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:42:45 dc2oss05 kernel: : LustreError: 29895:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8809cabefc00 x1461680324745001/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037838 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:42:45 dc2oss05 kernel: : LustreError: 29895:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 11:42:45 dc2oss05 kernel: : Lustre: dc2-OST002f: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:42:45 dc2oss05 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 11:42:47 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394037767 last 1394037767 Mar 5 11:42:47 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 11:42:55 dc2oss16 kernel: : Lustre: 17166:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394037775 last 1394037775 Mar 5 11:42:55 dc2oss16 kernel: : Lustre: 17166:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 11:42:59 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from 025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp t0 exp ffff880f961fbc00 cur 1394037779 last 1394037779 Mar 5 11:42:59 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 137 previous similar messages Mar 5 11:43:28 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:43:28 dc2oss03 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 11:43:28 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394037808 last 1394037808 Mar 5 11:43:28 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 11:43:33 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394037813 last 1394037813 Mar 5 11:43:33 dc2oss11 kernel: : Lustre: 16228:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 11:43:28 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 11:43:29 dc2oss07 kernel: : LustreError: 6201:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88009b16f000 x1461680324744899/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 1 to 0 dl 1394037937 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:43:29 dc2oss07 kernel: : LustreError: 6201:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 11:43:29 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:43:29 dc2oss07 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 11:43:51 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0067_UUID' is not available for connect (no target) Mar 5 11:43:51 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:43:53 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc10e800 cur 1394037833 last 1394037833 Mar 5 11:43:53 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 151 previous similar messages Mar 5 11:44:01 dc2oss16 kernel: : Lustre: dc2-OST00a1: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:44:01 dc2oss16 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 11:44:00 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:44:00 dc2oss04 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 11:44:15 dc2oss02 kernel: : Lustre: dc2-OST0013: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:44:15 dc2oss02 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 11:44:15 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0013: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817e9d40c00 cur 1394037855 last 1394037855 Mar 5 11:44:15 dc2oss02 kernel: : Lustre: 16322:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:44:23 dc2oss05 kernel: : Lustre: 15991:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394037863 last 1394037863 Mar 5 11:44:23 dc2oss05 kernel: : Lustre: 15991:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 192 previous similar messages Mar 5 11:44:37 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:44:37 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 11:44:51 dc2oss08 kernel: : Lustre: Service thread pid 27666 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 11:44:51 dc2oss08 kernel: : LustreError: dumping log to /tmp/lustre-log.1394037891.27666 Mar 5 11:44:51 dc2oss08 kernel: : Lustre: Service thread pid 27748 was inactive for 200.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 11:44:51 dc2oss08 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 11:45:10 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0078_UUID' is not available for connect (no target) Mar 5 11:45:10 dc2oss11 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 11:45:21 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:45:21 dc2oss14 kernel: : Lustre: Skipped 135 previous similar messages Mar 5 11:45:28 dc2oss11 kernel: : Lustre: dc2-OST006c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:45:28 dc2oss11 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 11:45:27 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:45:27 dc2oss13 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 11:45:18 dc2mds01 kernel: : Lustre: 14560:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb00:0x17:0x0]] [valid 68719476736] Mar 5 11:45:18 dc2mds01 kernel: : Lustre: 14560:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 56 previous similar messages Mar 5 11:45:36 dc2oss01 kernel: : Lustre: dc2-OST000a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:45:36 dc2oss01 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 11:45:36 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181877a000 cur 1394037936 last 1394037936 Mar 5 11:45:36 dc2oss01 kernel: : Lustre: 15074:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 11:45:50 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:45:50 dc2oss09 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 11:46:18 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 11:46:18 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 11:46:27 dc2oss15 kernel: : Lustre: dc2-OST0097: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:46:27 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:46:27 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 11:46:27 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 11:47:37 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0045_UUID' is not available for connect (no target) Mar 5 11:47:37 dc2oss08 kernel: : LustreError: Skipped 45 previous similar messages Mar 5 11:47:39 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:47:39 dc2oss06 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 11:48:09 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:48:09 dc2oss05 kernel: : Lustre: Skipped 205 previous similar messages Mar 5 11:48:10 dc2oss12 kernel: : Lustre: dc2-OST0077: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:48:10 dc2oss12 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 11:48:29 dc2oss07 kernel: : Lustre: dc2-OST0049: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:48:29 dc2oss07 kernel: : Lustre: Skipped 157 previous similar messages Mar 5 11:48:29 dc2oss07 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0049: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fc5f1800 cur 1394038109 last 1394038109 Mar 5 11:48:29 dc2oss07 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 154 previous similar messages Mar 5 11:49:10 dc2oss14 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 110s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST008e_UUID lock: ffff8817f139d480/0xe6b6f4ac8ab961b7 lrc: 3/0,0 mode: PR/PR res: 21850653/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c58ded4ec expref: 398 pid: 15630 timeout 9053383624 Mar 5 11:49:27 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 11:49:27 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 11:49:38 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001f_UUID' is not available for connect (no target) Mar 5 11:49:38 dc2oss04 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 11:49:52 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 11:49:52 dc2oss12 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 11:49:57 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 11:49:57 dc2oss15 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 11:50:06 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:50:06 dc2oss10 kernel: : Lustre: Skipped 153 previous similar messages Mar 5 11:50:42 dc2oss09 kernel: : Lustre: 16461:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394038242 last 1394038242 Mar 5 11:50:42 dc2oss09 kernel: : Lustre: 16461:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 11:50:51 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394038251 last 1394038251 Mar 5 11:50:51 dc2oss12 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 11:50:53 dc2oss11 kernel: : Lustre: 25363:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 11:50:58 dc2oss11 kernel: : LustreError: 8768:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd3dc0000 x1442912236948424/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394038258 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:50:58 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:51:02 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 11:51:02 dc2oss11 kernel: : LustreError: 14709:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc7785c00 x1442912236950316/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394038454 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:51:02 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 11:51:03 dc2oss11 kernel: : LustreError: 25352:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880066ceec00 x1442912236956126/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394038322 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:51:03 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:03 dc2xfer01 kernel: : Lustre: 2781:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:03 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:03 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:07 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:07 dc2xfer03 kernel: : Lustre: 2642:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:19 dc2xfer02 kernel: : Lustre: 2700:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:19 dc2xfer02 kernel: : Lustre: 2700:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 11:51:26 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 11:51:26 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 11:51:07 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 11:51:07 dc2oss01 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:51:17 dc2oss13 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394038277 last 1394038277 Mar 5 11:51:17 dc2oss13 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0088: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c8c7c00 cur 1394038277 last 1394038277 Mar 5 11:51:17 dc2oss13 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 11:51:38 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 11:51:38 dc2oss06 kernel: : LustreError: Skipped 52 previous similar messages Mar 5 11:51:42 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 11:51:42 dc2oss14 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 11:51:50 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394038310 last 1394038310 Mar 5 11:51:50 dc2oss15 kernel: : Lustre: 15976:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 11:51:47 dc2oss04 kernel: : Lustre: 16563:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394038307 last 1394038307 Mar 5 11:51:47 dc2oss04 kernel: : Lustre: 16563:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 11:52:23 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t36701401 exp ffff880c04583800 cur 1394038343 last 1394038343 Mar 5 11:52:23 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 11:52:36 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 11:52:36 dc2oss13 kernel: : LustreError: Skipped 39 previous similar messages Mar 5 11:52:39 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 11:52:39 dc2oss07 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:53:00 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 11:53:00 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 11:52:44 dc2oss08 kernel: : Lustre: dc2-OST004c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:52:44 dc2oss08 kernel: : Lustre: dc2-OST004f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:52:44 dc2oss08 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 11:52:44 dc2oss08 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 11:52:46 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 11:52:46 dc2oss16 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:52:52 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394038372 last 1394038372 Mar 5 11:52:52 dc2oss06 kernel: : Lustre: 14404:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 11:53:03 dc2oss16 kernel: : Lustre: 17103:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394038383 last 1394038383 Mar 5 11:53:03 dc2oss16 kernel: : Lustre: 17103:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 88 previous similar messages Mar 5 11:53:06 dc2oss08 kernel: : Lustre: 14580:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-95), not sending early reply Mar 5 11:53:06 dc2oss08 kernel: : Lustre: 14580:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Mar 5 11:53:11 dc2oss08 kernel: : LustreError: 27666:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 700+0s req@ffff880b9f6d6800 x1461680324744993/t0(0) o3->025d8915-d3b0-cfde-0c8c-221ff72be5fc@149.165.229.58@tcp:0/0 lens 448/400 e 3 to 0 dl 1394038391 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 11:53:11 dc2oss08 kernel: : Lustre: dc2-OST004b: Bulk IO read error with 025d8915-d3b0-cfde-0c8c-221ff72be5fc (at 149.165.229.58@tcp), client will retry: rc -110 Mar 5 11:53:11 dc2oss08 kernel: : Lustre: Service thread pid 27666 completed after 700.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 11:53:15 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394038395 last 1394038395 Mar 5 11:53:15 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 11:53:09 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:53:09 dc2mds01 kernel: : Lustre: Skipped 31 previous similar messages Mar 5 11:53:09 dc2mds01 kernel: : Lustre: 11179:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394038389 last 1394038389 Mar 5 11:53:09 dc2mds01 kernel: : Lustre: 11179:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages Mar 5 11:53:42 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:53:42 dc2oss03 kernel: : Lustre: Skipped 158 previous similar messages Mar 5 11:53:42 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394038422 last 1394038422 Mar 5 11:53:42 dc2oss03 kernel: : Lustre: 16905:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 157 previous similar messages Mar 5 11:53:43 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0xa5:0x0]] [valid 68719476736] Mar 5 11:53:58 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f6a2400 cur 1394038438 last 1394038438 Mar 5 11:53:58 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:54:10 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:54:10 dc2oss16 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 11:54:07 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:54:07 dc2oss04 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 11:54:21 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394038461 last 1394038461 Mar 5 11:54:21 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 11:54:26 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:54:26 dc2oss02 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 11:54:26 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394038466 last 1394038466 Mar 5 11:54:26 dc2oss02 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 11:54:24 dc2oss05 kernel: : Lustre: 14589:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394038464 last 1394038464 Mar 5 11:54:24 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394038464 last 1394038464 Mar 5 11:54:24 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 11:54:24 dc2oss05 kernel: : Lustre: 14589:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 115 previous similar messages Mar 5 11:55:00 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 942560, left: 942560, last alive is 31 secs ago Mar 5 11:55:00 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 942560, left: 942560, last alive is 31 secs ago Mar 5 11:55:12 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 11:55:12 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:55:14 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 11:55:14 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 11:55:27 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:55:27 dc2oss14 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 11:55:30 dc2oss13 kernel: : Lustre: dc2-OST0088: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:55:30 dc2oss13 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 11:55:36 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:55:36 dc2oss11 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 11:55:38 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) Mar 5 11:55:38 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 11:55:34 dc2mds01 kernel: : Lustre: 11179:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0f:0x188d:0x0]] [valid 68719476736] Mar 5 11:55:34 dc2mds01 kernel: : Lustre: 11179:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 84 previous similar messages Mar 5 11:55:38 dc2oss01 kernel: : Lustre: dc2-OST0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:55:38 dc2oss01 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 11:55:38 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c003ae000 cur 1394038538 last 1394038538 Mar 5 11:55:38 dc2oss01 kernel: : Lustre: 15583:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 11:55:51 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:55:51 dc2oss09 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 11:56:28 dc2oss15 kernel: : Lustre: dc2-OST0098: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:56:28 dc2oss15 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 11:56:34 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 11:56:34 dc2oss10 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 11:57:05 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 134752, left: 134752, last alive is 22 secs ago Mar 5 11:57:05 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 134752, left: 134752, last alive is 22 secs ago Mar 5 11:57:52 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 11:57:52 dc2oss08 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 11:57:58 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:57:58 dc2oss06 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 11:58:11 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:58:11 dc2oss05 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 11:58:16 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:58:16 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 11:58:34 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 23 secs ago Mar 5 11:58:34 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 23 secs ago Mar 5 11:58:39 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 11:58:39 dc2oss07 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 11:58:39 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394038719 last 1394038719 Mar 5 11:58:39 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 11:59:47 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 11:59:47 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:00:05 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 12:00:05 dc2oss12 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 12:00:06 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 12:00:06 dc2oss15 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:00:22 dc2oss10 kernel: : Lustre: dc2-OST0066: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:00:22 dc2oss10 kernel: : Lustre: Skipped 76 previous similar messages Mar 5 12:00:36 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001e_UUID' is not available for connect (no target) Mar 5 12:00:36 dc2oss04 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:00:51 dc2oss09 kernel: : Lustre: 16360:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25e45c00 cur 1394038851 last 1394038851 Mar 5 12:00:51 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394038851 last 1394038851 Mar 5 12:00:51 dc2oss09 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 12:00:53 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 663392, left: 663392, last alive is 1 secs ago Mar 5 12:00:53 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 663392, left: 663392, last alive is 1 secs ago Mar 5 12:01:14 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394038874 last 1394038874 Mar 5 12:01:14 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 12:01:19 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394038879 last 1394038879 Mar 5 12:01:19 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 12:01:47 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0002_UUID' is not available for connect (no target) Mar 5 12:01:47 dc2oss02 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:01:44 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 12:01:44 dc2oss06 kernel: : LustreError: Skipped 37 previous similar messages Mar 5 12:01:51 dc2oss15 kernel: : Lustre: 16356:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394038911 last 1394038911 Mar 5 12:01:51 dc2oss15 kernel: : Lustre: 16356:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 12:01:48 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394038908 last 1394038908 Mar 5 12:01:48 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 12:02:01 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0086_UUID' is not available for connect (no target) Mar 5 12:02:01 dc2oss14 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:01:55 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0012_UUID' is not available for connect (no target) Mar 5 12:01:55 dc2oss01 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 12:02:23 dc2oss14 kernel: : Lustre: 15506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394038943 last 1394038943 Mar 5 12:02:23 dc2oss14 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37972127 exp ffff880c08fefc00 cur 1394038943 last 1394038943 Mar 5 12:02:23 dc2oss14 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 12:02:44 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 12:02:44 dc2oss07 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 12:02:42 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 12:02:42 dc2oss13 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:03:07 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0027_UUID' is not available for connect (no target) Mar 5 12:03:07 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:02:57 dc2oss08 kernel: : Lustre: dc2-OST0053: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:02:57 dc2oss08 kernel: : Lustre: Skipped 64 previous similar messages Mar 5 12:02:54 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394038974 last 1394038974 Mar 5 12:02:54 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 12:02:55 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 12:02:55 dc2oss16 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:03:18 dc2oss11 kernel: : Lustre: Service thread pid 25327 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 12:03:18 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 12:03:18 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394038998.25327 Mar 5 12:03:19 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394038999 last 1394038999 Mar 5 12:03:19 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 12:03:22 dc2oss08 kernel: : Lustre: 14456:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394039002 last 1394039002 Mar 5 12:03:22 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182beb1800 cur 1394039002 last 1394039002 Mar 5 12:03:22 dc2oss08 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 73 previous similar messages Mar 5 12:03:21 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 12:03:21 dc2mds01 kernel: : Lustre: Skipped 36 previous similar messages Mar 5 12:03:21 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394039001 last 1394039001 Mar 5 12:03:21 dc2mds01 kernel: : Lustre: 17953:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 38 previous similar messages Mar 5 12:04:08 dc2oss03 kernel: : Lustre: dc2-OST0015: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:04:08 dc2oss03 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 12:04:08 dc2oss03 kernel: : Lustre: 17051:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cf19400 cur 1394039048 last 1394039048 Mar 5 12:04:08 dc2oss03 kernel: : Lustre: 17051:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 12:04:09 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881825b13400 cur 1394039049 last 1394039049 Mar 5 12:04:09 dc2oss11 kernel: : Lustre: 15481:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 12:04:12 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:04:12 dc2oss16 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 12:04:26 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:04:26 dc2oss02 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 12:04:26 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394039066 last 1394039066 Mar 5 12:04:26 dc2oss02 kernel: : Lustre: 16371:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 12:04:12 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:04:12 dc2oss04 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 12:04:32 dc2oss05 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394039072 last 1394039072 Mar 5 12:04:32 dc2oss05 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 12:04:35 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394039075 last 1394039075 Mar 5 12:04:35 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 12:05:33 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 12:05:33 dc2oss09 kernel: : LustreError: Skipped 12 previous similar messages Mar 5 12:05:35 dc2oss14 kernel: : Lustre: dc2-OST0092: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:05:35 dc2oss14 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:05:39 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:05:39 dc2oss11 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 12:05:42 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 12:05:42 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 12:05:47 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:05:47 dc2oss13 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 12:05:54 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:05:54 dc2oss09 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 12:05:45 dc2oss01 kernel: : Lustre: dc2-OST0008: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:05:45 dc2oss01 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 12:05:45 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0008: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6c77800 cur 1394039145 last 1394039145 Mar 5 12:05:45 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 12:05:50 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f7b2:0x29:0x0]] [valid 68719476736] Mar 5 12:05:50 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 71 previous similar messages Mar 5 12:06:29 dc2oss11 kernel: : Lustre: 25413:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:06:33 dc2oss11 kernel: : Lustre: 8805:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:06:34 dc2oss11 kernel: : LustreError: 8707:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd3b02c00 x1442912236959584/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039194 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:06:34 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:06:34 dc2oss11 kernel: : Lustre: Skipped 5 previous similar messages Mar 5 12:06:30 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:06:30 dc2oss15 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 12:06:38 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 12:06:38 dc2oss10 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:06:38 dc2oss11 kernel: : LustreError: 8750:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880a1bd2fc00 x1442912236959592/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039198 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:06:38 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:06:41 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 7 active RPCs Mar 5 12:06:41 dc2oss11 kernel: : LustreError: 25395:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880e18606050 x1442912236962335/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039330 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:06:41 dc2oss11 kernel: : LustreError: 25395:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 12:06:41 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:06:42 dc2oss11 kernel: : LustreError: 8732:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd914ac00 x1442912236974132/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039953 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:06:42 dc2oss11 kernel: : LustreError: 8732:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 12:06:42 dc2oss11 kernel: : Lustre: Service thread pid 25327 completed after 404.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 12:06:28 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 12:06:28 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 31 previous similar messages Mar 5 12:06:42 dc2xfer01 kernel: : Lustre: 2782:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:06:42 dc2xfer01 kernel: : Lustre: 2782:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:06:46 dc2xfer03 kernel: : Lustre: 2647:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:06:46 dc2xfer03 kernel: : Lustre: 2647:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:07:59 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:07:59 dc2oss06 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 12:08:03 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 995264, left: 995264, last alive is 10 secs ago Mar 5 12:08:03 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 995264, left: 995264, last alive is 10 secs ago Mar 5 12:08:13 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 12:08:13 dc2oss08 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 12:08:22 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:08:22 dc2oss05 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 12:08:20 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:08:20 dc2oss12 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 12:08:41 dc2oss07 kernel: : Lustre: dc2-OST003f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:08:41 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 12:08:41 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182c523800 cur 1394039321 last 1394039321 Mar 5 12:08:41 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 12:08:38 dc2oss11 kernel: : LustreError: 25461:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 100+0s req@ffff8801c09d5000 x1442912236974367/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039318 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 12:08:38 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:08:38 dc2oss11 kernel: : Lustre: Skipped 6 previous similar messages Mar 5 12:09:52 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0039_UUID' is not available for connect (no target) Mar 5 12:09:52 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:10:26 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:10:26 dc2oss10 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:10:44 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 12:10:44 dc2oss15 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:10:51 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394039451 last 1394039451 Mar 5 12:10:51 dc2oss09 kernel: : Lustre: 16310:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 12:11:16 dc2oss12 kernel: : Lustre: 14476:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394039476 last 1394039476 Mar 5 12:11:16 dc2oss12 kernel: : Lustre: 14476:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 12:11:20 dc2oss13 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0084: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806390c00 cur 1394039480 last 1394039480 Mar 5 12:11:20 dc2oss13 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 12:11:24 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 475232, left: 475232, last alive is 1 secs ago Mar 5 12:11:24 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 475232, left: 475232, last alive is 1 secs ago Mar 5 12:11:46 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 12:11:46 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:11:58 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 12:11:58 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 12:11:51 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394039511 last 1394039511 Mar 5 12:11:51 dc2oss15 kernel: : Lustre: 14538:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 12:11:48 dc2oss04 kernel: : Lustre: 16478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bffed2000 cur 1394039508 last 1394039508 Mar 5 12:11:48 dc2oss04 kernel: : Lustre: 16478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 12:12:07 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 12:12:07 dc2oss04 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 12:12:17 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 12:12:17 dc2oss06 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 12:12:25 dc2oss14 kernel: : Lustre: 14587:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394039545 last 1394039545 Mar 5 12:12:25 dc2oss14 kernel: : Lustre: 14587:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 12:12:40 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0087_UUID' is not available for connect (no target) Mar 5 12:12:40 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:12:59 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fdb1bc00 cur 1394039579 last 1394039579 Mar 5 12:12:59 dc2oss06 kernel: : Lustre: 14392:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 12:13:03 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009c_UUID' is not available for connect (no target) Mar 5 12:13:03 dc2oss16 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:13:15 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008c_UUID' is not available for connect (no target) Mar 5 12:13:15 dc2oss13 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:13:22 dc2oss16 kernel: : Lustre: 17103:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394039602 last 1394039602 Mar 5 12:13:22 dc2oss16 kernel: : Lustre: 17103:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 163 previous similar messages Mar 5 12:13:42 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 12:13:42 dc2oss03 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 12:13:23 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0013_UUID' is not available for connect (no target) Mar 5 12:13:23 dc2oss01 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:13:33 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 12:13:33 dc2mds01 kernel: : Lustre: Skipped 33 previous similar messages Mar 5 12:13:33 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394039613 last 1394039613 Mar 5 12:13:33 dc2mds01 kernel: : Lustre: 9750:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 47 previous similar messages Mar 5 12:14:22 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:14:22 dc2oss03 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 12:14:22 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394039662 last 1394039662 Mar 5 12:14:22 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 170 previous similar messages Mar 5 12:14:17 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:14:17 dc2oss16 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 12:14:17 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:14:17 dc2oss04 kernel: : Lustre: Skipped 72 previous similar messages Mar 5 12:14:48 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:14:48 dc2oss02 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 12:14:48 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c016d4800 cur 1394039688 last 1394039688 Mar 5 12:14:48 dc2oss02 kernel: : Lustre: 16208:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 149 previous similar messages Mar 5 12:14:38 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394039678 last 1394039678 Mar 5 12:14:38 dc2oss10 kernel: : Lustre: 16326:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 157 previous similar messages Mar 5 12:14:38 dc2oss05 kernel: : Lustre: 16035:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394039678 last 1394039678 Mar 5 12:14:38 dc2oss05 kernel: : Lustre: 16035:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 178 previous similar messages Mar 5 12:14:59 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394039699 last 1394039699 Mar 5 12:14:59 dc2oss11 kernel: : Lustre: 16285:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e5c2c00 cur 1394039699 last 1394039699 Mar 5 12:14:59 dc2oss11 kernel: : Lustre: 16285:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 143 previous similar messages Mar 5 12:14:59 dc2oss11 kernel: : Lustre: 14537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message Mar 5 12:15:28 dc2oss08 kernel: : Lustre: dc2-OST004d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:28 dc2oss08 kernel: : Lustre: Skipped 66 previous similar messages Mar 5 12:15:28 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394039728 last 1394039728 Mar 5 12:15:28 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 132 previous similar messages Mar 5 12:15:35 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:35 dc2oss14 kernel: : Lustre: Skipped 87 previous similar messages Mar 5 12:15:48 dc2oss13 kernel: : Lustre: dc2-OST007f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:48 dc2oss13 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 12:15:48 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:48 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:48 dc2oss11 kernel: : Lustre: Skipped 67 previous similar messages Mar 5 12:15:48 dc2oss11 kernel: : Lustre: Skipped 67 previous similar messages Mar 5 12:15:56 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0064_UUID' is not available for connect (no target) Mar 5 12:15:56 dc2oss09 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:16:05 dc2oss09 kernel: : Lustre: dc2-OST0054: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:16:05 dc2oss09 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 12:15:50 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:15:50 dc2oss01 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 12:15:50 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394039750 last 1394039750 Mar 5 12:15:50 dc2oss01 kernel: : Lustre: 15462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 158 previous similar messages Mar 5 12:16:16 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004e_UUID' is not available for connect (no target) Mar 5 12:16:16 dc2oss07 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 12:16:20 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 12:16:20 dc2oss11 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 12:16:38 dc2oss15 kernel: : Lustre: dc2-OST009b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:16:38 dc2oss15 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 12:16:50 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0058_UUID' is not available for connect (no target) Mar 5 12:16:50 dc2oss10 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:17:38 dc2oss08 kernel: : Lustre: 7147:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394039848/real 1394039857] req@ffff8815cc2d0c00 x1456766190353074/t0(0) o105->dc2-OST004b@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394039858 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 12:17:38 dc2oss08 kernel: : LustreError: 138-a: dc2-OST004b: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 12:18:03 dc2oss06 kernel: : Lustre: dc2-OST0038: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:18:03 dc2oss06 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 12:18:22 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:18:22 dc2oss05 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:18:31 dc2oss12 kernel: : Lustre: dc2-OST007b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:18:31 dc2oss12 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 12:18:31 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0041_UUID' is not available for connect (no target) Mar 5 12:18:31 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:18:23 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0c:0x1d:0x0]] [valid 68719476736] Mar 5 12:18:23 dc2mds01 kernel: : Lustre: 1387:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 55 previous similar messages Mar 5 12:18:42 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:18:42 dc2oss07 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 12:18:42 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394039922 last 1394039922 Mar 5 12:18:42 dc2oss07 kernel: : Lustre: 14433:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 178 previous similar messages Mar 5 12:19:33 dc2oss11 kernel: : Lustre: 25414:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:19:34 dc2oss11 kernel: : Lustre: 25454:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:19:38 dc2oss11 kernel: : LustreError: 8768:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd3809400 x1442912236974370/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039978 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:19:38 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:19:39 dc2oss11 kernel: : LustreError: 14655:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff8817fafe7850 x1442912236974592/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394039979 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:19:39 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:19:44 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 12:19:44 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880170a50000 x1442912236982139/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394040063 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:19:44 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages Mar 5 12:19:44 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:19:45 dc2oss11 kernel: : LustreError: 14618:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd71e7c00 x1442912236982375/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394040737 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:19:45 dc2oss11 kernel: : LustreError: 25413:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff88161bd22050 x1442912236976703/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394040179 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:19:45 dc2oss11 kernel: : LustreError: 25413:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 12:19:46 dc2xfer03 kernel: : Lustre: 2646:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:19:46 dc2xfer03 kernel: : Lustre: 2646:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:19:48 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:19:48 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:20:09 149.165.238.1 kernel: kernel: [4314381.260030] Lustre: 23964:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105830005 sent from dc2-OST0053-osc-ffff880956851000 to NID 10.10.0.8@o2ib 19s ago has timed out (19s prior to deadline). Mar 5 12:20:09 149.165.238.1 kernel: kernel: [4314381.260042] Lustre: dc2-OST0053-osc-ffff880956851000: Connection to service dc2-OST0053 via nid 10.10.0.8@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 12:20:09 149.165.238.1 kernel: kernel: [4314381.313542] Lustre: dc2-OST0053-osc-ffff880956851000: Connection restored to service dc2-OST0053 using nid 10.10.0.8@o2ib. Mar 5 12:20:09 149.165.238.1 kernel: kernel: [4314381.313551] Lustre: Server dc2-OST0053_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 12:20:33 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:20:33 dc2oss10 kernel: : Lustre: Skipped 76 previous similar messages Mar 5 12:20:38 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 12:20:38 dc2mds01 kernel: : LustreError: 18114:0:(obd_class.h:503:obd_set_info_async()) Skipped 26 previous similar messages Mar 5 12:20:52 dc2oss09 kernel: : Lustre: 16136:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394040052 last 1394040052 Mar 5 12:20:52 dc2oss09 kernel: : Lustre: 16136:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 195 previous similar messages Mar 5 12:21:01 dc2oss10 kernel: : Lustre: dc2-OST0061: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 12:21:01 dc2oss10 kernel: : LustreError: 8678:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88095729c050 x1461679863152186/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394040108 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:21:01 dc2oss10 kernel: : LustreError: 8678:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 12:21:01 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 12:21:02 dc2oss10 kernel: : LustreError: 8870:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88018029d000 x1461679863152423/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 1 to 0 dl 1394040111 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:21:02 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 12:21:28 dc2oss03 kernel: : Lustre: dc2-OST001a: Client eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 12:21:29 dc2oss03 kernel: : LustreError: 20915:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806b7448000 x1461679863152475/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394040102 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:21:29 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 12:21:29 dc2oss03 kernel: : LustreError: 16755:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8802fc8e3c00 x1461679863152455/t0(0) o3->eb673071-d382-00d2-37ca-c8916eb31cd5@149.165.229.63@tcp:0/0 lens 448/400 e 0 to 0 dl 1394040100 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:21:29 dc2oss03 kernel: : Lustre: dc2-OST001a: Bulk IO read error with eb673071-d382-00d2-37ca-c8916eb31cd5 (at 149.165.229.63@tcp), client will retry: rc -110 Mar 5 12:21:11 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 12:21:11 dc2oss15 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:21:18 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394040078 last 1394040078 Mar 5 12:21:18 dc2oss12 kernel: : Lustre: 15273:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 168 previous similar messages Mar 5 12:21:26 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0039_UUID' is not available for connect (no target) Mar 5 12:21:26 dc2oss05 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:21:28 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394040088 last 1394040088 Mar 5 12:21:28 dc2oss13 kernel: : Lustre: 15952:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 201 previous similar messages Mar 5 12:21:19 dc2mds01 kernel: : LustreError: 9399:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232067270268, new: 232067270267 replay: 0. see LU-617. Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.337816] LustreError: 11-0: an error occurred while communicating with 10.10.0.8@o2ib. The ldlm_enqueue operation failed with -107 Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.337825] Lustre: dc2-OST0053-osc-ffff880956851000: Connection to service dc2-OST0053 via nid 10.10.0.8@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.339493] LustreError: 167-0: This client was evicted by dc2-OST0053; in progress operations using this service will fail. Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.339531] Lustre: Server dc2-OST0053_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.339573] LustreError: 26389:0:(file.c:998:ll_glimpse_size()) obd_enqueue returned rc -4, returning -EIO Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.339620] LustreError: 23632:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff880dba6b8400 x1461669105850510/t0 o4->dc2-OST0053_UUID@10.10.0.8@o2ib:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0 Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.339626] LustreError: 23632:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 6 previous similar messages Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.345997] LustreError: 26391:0:(file.c:998:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.375986] LustreError: 26390:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Namespace dc2-OST0053-osc-ffff880956851000 resource refcount nonzero (1) after lock cleanup; forcing cleanup. Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.375996] LustreError: 26390:0:(ldlm_resource.c:554:ldlm_namespace_cleanup()) Skipped 25 previous similar messages Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.376001] LustreError: 26390:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Resource: ffff880e4e12fc80 (23198081/0/0/0) (rc: 1) Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.376004] LustreError: 26390:0:(ldlm_resource.c:559:ldlm_namespace_cleanup()) Skipped 25 previous similar messages Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.376385] LustreError: 23964:0:(llite_lib.c:1489:ll_setattr_do_truncate()) extent unlock failed: err=-5, unlock method =1 Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.377792] LustreError: 23964:0:(namei.c:1210:ll_objects_destroy()) obd destroy objid 0x31d9@0x202f0fb0f error -5 Mar 5 12:21:50 149.165.238.1 kernel: kernel: [4314482.422957] Lustre: dc2-OST0053-osc-ffff880956851000: Connection restored to service dc2-OST0053 using nid 10.10.0.8@o2ib. Mar 5 12:21:49 dc2oss08 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 119s: evicting client at 149.165.238.1@tcp ns: filter-dc2-OST0053_UUID lock: ffff88173bc25000/0x9b0087050683627 lrc: 3/0,0 mode: PW/PW res: 23198081/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xe213f1fec4ead916 expref: 231 pid: 14548 timeout 9055344889 Mar 5 12:21:53 dc2oss08 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 123s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST0052_UUID lock: ffff88173bc25d80/0x9b0087050683620 lrc: 3/0,0 mode: PR/PR res: 23060169/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c5903fc72 expref: 228 pid: 14548 timeout 9055348475 Mar 5 12:21:51 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 12:21:51 dc2oss12 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:21:49 dc2oss04 kernel: : Lustre: 16091:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394040109 last 1394040109 Mar 5 12:21:49 dc2oss04 kernel: : Lustre: 16091:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 138 previous similar messages Mar 5 12:21:53 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0099: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf5c5dc00 cur 1394040113 last 1394040113 Mar 5 12:21:53 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 195 previous similar messages Mar 5 12:22:13 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 12:22:13 dc2oss04 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:22:25 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d7ab800 cur 1394040145 last 1394040145 Mar 5 12:22:25 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 166 previous similar messages Mar 5 12:22:31 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 12:22:31 dc2oss06 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 12:22:55 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0007_UUID' is not available for connect (no target) Mar 5 12:22:55 dc2oss02 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 12:22:54 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 12:22:54 dc2oss14 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:22:59 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 144064, left: 144064, last alive is 24 secs ago Mar 5 12:22:59 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 144064, left: 144064, last alive is 24 secs ago Mar 5 12:23:15 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 12:23:15 dc2oss16 kernel: : LustreError: Skipped 36 previous similar messages Mar 5 12:23:22 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394040202 last 1394040202 Mar 5 12:23:22 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 12:23:17 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008b_UUID' is not available for connect (no target) Mar 5 12:23:17 dc2oss13 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:23:19 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eb3e800 cur 1394040199 last 1394040199 Mar 5 12:23:19 dc2oss06 kernel: : Lustre: 15878:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 139 previous similar messages Mar 5 12:23:46 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0026_UUID' is not available for connect (no target) Mar 5 12:23:46 dc2oss03 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 12:24:28 dc2oss03 kernel: : Lustre: dc2-OST0018: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:28 dc2oss03 kernel: : Lustre: Skipped 137 previous similar messages Mar 5 12:24:28 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0cfc2400 cur 1394040268 last 1394040268 Mar 5 12:24:28 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 136 previous similar messages Mar 5 12:24:03 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 12:24:03 dc2oss01 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:24:18 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:18 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:18 dc2oss16 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 12:24:18 dc2oss16 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 12:24:06 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:06 dc2mds01 kernel: : Lustre: Skipped 20 previous similar messages Mar 5 12:24:06 dc2mds01 kernel: : Lustre: 1368:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394040246 last 1394040246 Mar 5 12:24:06 dc2mds01 kernel: : Lustre: 1368:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 20 previous similar messages Mar 5 12:24:23 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:23 dc2oss04 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 12:24:55 dc2oss02 kernel: : Lustre: dc2-OST0010: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:24:55 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb067000 cur 1394040295 last 1394040295 Mar 5 12:24:55 dc2oss02 kernel: : Lustre: Skipped 93 previous similar messages Mar 5 12:24:55 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 93 previous similar messages Mar 5 12:24:39 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0fec3800 cur 1394040279 last 1394040279 Mar 5 12:24:39 dc2oss10 kernel: : Lustre: 16464:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 12:24:42 dc2oss05 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394040282 last 1394040282 Mar 5 12:24:42 dc2oss05 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 12:25:24 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394040324 last 1394040324 Mar 5 12:25:24 dc2oss11 kernel: : Lustre: 15510:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 12:25:36 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:25:36 dc2oss08 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 12:25:36 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394040335 last 1394040335 Mar 5 12:25:36 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 12:25:39 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:25:39 dc2oss14 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 12:25:52 dc2oss13 kernel: : Lustre: dc2-OST0085: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:25:52 dc2oss13 kernel: : Lustre: Skipped 135 previous similar messages Mar 5 12:25:57 dc2oss11 kernel: : Lustre: dc2-OST0069: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:25:57 dc2oss11 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 12:26:05 dc2oss09 kernel: : Lustre: dc2-OST005b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:26:05 dc2oss09 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 12:26:02 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:26:02 dc2oss01 kernel: : Lustre: Skipped 79 previous similar messages Mar 5 12:26:02 dc2oss01 kernel: : Lustre: 15376:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394040362 last 1394040362 Mar 5 12:26:02 dc2oss01 kernel: : Lustre: 15376:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 12:26:17 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 12:26:17 dc2oss07 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 12:26:39 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:26:39 dc2oss15 kernel: : Lustre: Skipped 117 previous similar messages Mar 5 12:27:11 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 520736, left: 520736, last alive is 43 secs ago Mar 5 12:27:11 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 520736, left: 520736, last alive is 43 secs ago Mar 5 12:27:17 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 12:27:17 dc2oss11 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 12:27:27 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 12:27:27 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:28:16 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:28:16 dc2oss06 kernel: : Lustre: Skipped 91 previous similar messages Mar 5 12:28:26 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0067_UUID' is not available for connect (no target) Mar 5 12:28:26 dc2oss09 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:28:27 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:28:27 dc2oss05 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 12:28:35 dc2oss12 kernel: : Lustre: dc2-OST0078: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:28:35 dc2oss12 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 12:29:04 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:29:04 dc2oss07 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 12:29:04 dc2oss07 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394040544 last 1394040544 Mar 5 12:29:04 dc2oss07 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 12:29:10 dc2mds01 kernel: : Lustre: 11195:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f797:0x1574:0x0]] [valid 68719476736] Mar 5 12:29:10 dc2mds01 kernel: : Lustre: 11195:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 45 previous similar messages Mar 5 12:29:42 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 12:29:42 dc2oss08 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:30:08 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 251232, left: 251232, last alive is 5 secs ago Mar 5 12:30:08 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 251232, left: 251232, last alive is 5 secs ago Mar 5 12:31:22 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a3_UUID' is not available for connect (no target) Mar 5 12:31:22 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:31:23 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:31:23 dc2oss10 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 12:31:16 dc2oss09 kernel: : Lustre: 14539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394040676 last 1394040676 Mar 5 12:31:16 dc2oss09 kernel: : Lustre: 14539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 12:31:24 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8f4dc00 cur 1394040684 last 1394040684 Mar 5 12:31:24 dc2oss12 kernel: : Lustre: 14475:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 12:31:35 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 12:31:35 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:31:43 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827468c00 cur 1394040703 last 1394040703 Mar 5 12:31:43 dc2oss13 kernel: : Lustre: 15909:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 12:31:49 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394040709 last 1394040709 Mar 5 12:31:49 dc2oss04 kernel: : Lustre: 16361:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 12:31:56 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803441800 cur 1394040716 last 1394040716 Mar 5 12:31:56 dc2oss15 kernel: : Lustre: 15797:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 12:31:53 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 12:31:53 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 20 previous similar messages Mar 5 12:32:33 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818282a4c00 cur 1394040753 last 1394040753 Mar 5 12:32:33 dc2oss14 kernel: : Lustre: 15662:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 12:32:56 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0006_UUID' is not available for connect (no target) Mar 5 12:32:56 dc2oss02 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 12:33:11 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001e_UUID' is not available for connect (no target) Mar 5 12:33:11 dc2oss04 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 12:33:21 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394040801 last 1394040801 Mar 5 12:33:21 dc2oss06 kernel: : Lustre: 15936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 12:33:24 dc2oss16 kernel: : Lustre: 15767:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394040804 last 1394040804 Mar 5 12:33:24 dc2oss16 kernel: : Lustre: 15767:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 76 previous similar messages Mar 5 12:33:26 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 12:33:26 dc2oss14 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 12:33:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008f_UUID' is not available for connect (no target) Mar 5 12:33:33 dc2oss13 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:33:38 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0096_UUID' is not available for connect (no target) Mar 5 12:33:38 dc2oss16 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:34:30 dc2oss03 kernel: : Lustre: dc2-OST001f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:34:30 dc2oss03 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 12:34:30 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8415000 cur 1394040870 last 1394040870 Mar 5 12:34:30 dc2oss03 kernel: : Lustre: 17109:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 12:34:17 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0034_UUID' is not available for connect (no target) Mar 5 12:34:17 dc2oss06 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:34:25 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:34:25 dc2oss16 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 12:34:34 dc2oss11 kernel: : Lustre: 8743:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:34:28 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 12:34:28 dc2oss12 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 12:34:48 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 12:34:48 dc2oss03 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 12:34:30 dc2oss08 kernel: : Lustre: 15858:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394040863/real 1394040869] req@ffff881513856c00 x1456766190355578/t0(0) o106->dc2-OST0051@149.165.229.29@tcp:15/16 lens 296/232 e 0 to 1 dl 1394040870 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Mar 5 12:34:30 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:34:30 dc2oss04 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 12:34:55 dc2oss02 kernel: : Lustre: dc2-OST0014: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:34:55 dc2oss02 kernel: : Lustre: Skipped 70 previous similar messages Mar 5 12:34:55 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0014: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf2707400 cur 1394040895 last 1394040895 Mar 5 12:34:55 dc2oss02 kernel: : Lustre: 15917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 71 previous similar messages Mar 5 12:34:38 dc2oss11 kernel: : Lustre: 8638:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:34:39 dc2oss11 kernel: : LustreError: 14718:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff8817fa536850 x1442912236983826/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394040879 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:34:39 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:34:39 dc2oss11 kernel: : Lustre: Skipped 5 previous similar messages Mar 5 12:34:43 dc2oss11 kernel: : LustreError: 14684:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bd60c9400 x1442912236983828/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394040883 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:34:43 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:34:48 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 12:34:37 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:34:37 dc2mds01 kernel: : Lustre: Skipped 28 previous similar messages Mar 5 12:34:37 dc2mds01 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394040877 last 1394040877 Mar 5 12:34:37 dc2mds01 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 28 previous similar messages Mar 5 12:34:32 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000b_UUID' is not available for connect (no target) Mar 5 12:34:32 dc2oss01 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:34:44 dc2oss05 kernel: : Lustre: 15813:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827821800 cur 1394040884 last 1394040883 Mar 5 12:34:44 dc2oss05 kernel: : Lustre: 15813:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 12:34:44 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394040884 last 1394040883 Mar 5 12:34:44 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 12:34:49 dc2oss11 kernel: : LustreError: 25304:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bc71a3400 x1442912236991689/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041642 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:34:49 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:34:49 dc2oss11 kernel: : LustreError: 25304:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 12:34:49 dc2oss11 kernel: : LustreError: 25455:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd4d44400 x1442912236988354/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041305 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:34:49 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:34:49 dc2xfer01 kernel: : Lustre: 2784:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:34:49 dc2xfer02 kernel: : Lustre: 2704:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:34:49 dc2xfer02 kernel: : Lustre: 2704:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 12:35:37 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:35:37 dc2oss08 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 12:35:37 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394040937 last 1394040937 Mar 5 12:35:37 dc2oss08 kernel: : Lustre: 15865:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 12:35:43 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0073: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f211c400 cur 1394040943 last 1394040943 Mar 5 12:35:43 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 12:35:46 dc2oss14 kernel: : Lustre: dc2-OST008d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:35:46 dc2oss14 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 12:36:03 149.165.238.1 kernel: kernel: [4315335.660313] Lustre: 39435:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105911133 sent from dc2-OST00a4-osc-ffff880956851000 to NID 10.10.0.16@o2ib 7s ago has timed out (7s prior to deadline). Mar 5 12:36:03 149.165.238.1 kernel: kernel: [4315335.660326] Lustre: dc2-OST00a4-osc-ffff880956851000: Connection to service dc2-OST00a4 via nid 10.10.0.16@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 12:36:03 149.165.238.1 kernel: kernel: [4315335.660720] Lustre: dc2-OST00a4-osc-ffff880956851000: Connection restored to service dc2-OST00a4 using nid 10.10.0.16@o2ib. Mar 5 12:36:03 149.165.238.1 kernel: kernel: [4315335.660739] Lustre: Server dc2-OST00a4_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 12:35:58 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:35:58 dc2oss11 kernel: : Lustre: Skipped 76 previous similar messages Mar 5 12:36:06 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:36:06 dc2oss13 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 12:36:09 dc2oss01 kernel: : Lustre: dc2-OST0005: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:36:09 dc2oss01 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 12:36:09 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0005: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37715128 exp ffff880be6c96400 cur 1394040969 last 1394040969 Mar 5 12:36:09 dc2oss01 kernel: : Lustre: 15589:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 12:36:27 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:36:27 dc2oss09 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:36:41 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:36:41 dc2oss15 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 12:37:36 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005e_UUID' is not available for connect (no target) Mar 5 12:37:36 dc2oss10 kernel: : LustreError: Skipped 12 previous similar messages Mar 5 12:38:10 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004d_UUID' is not available for connect (no target) Mar 5 12:38:10 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:38:22 dc2oss06 kernel: : Lustre: dc2-OST0036: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:38:22 dc2oss06 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 12:38:32 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007c_UUID' is not available for connect (no target) Mar 5 12:38:32 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:38:31 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:38:31 dc2oss05 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 12:38:32 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 204864, left: 204864, last alive is 0 secs ago Mar 5 12:38:32 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 204864, left: 204864, last alive is 0 secs ago Mar 5 12:38:35 dc2oss12 kernel: : Lustre: dc2-OST0074: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:38:35 dc2oss12 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 12:38:43 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0062_UUID' is not available for connect (no target) Mar 5 12:38:43 dc2oss09 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 12:39:06 dc2oss07 kernel: : Lustre: dc2-OST0041: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:39:06 dc2oss07 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 12:39:06 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf0176400 cur 1394041146 last 1394041146 Mar 5 12:39:06 dc2oss07 kernel: : Lustre: 14382:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 12:39:18 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0e:0x1f:0x0]] [valid 68719476736] Mar 5 12:39:18 dc2mds01 kernel: : Lustre: 11151:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 42 previous similar messages Mar 5 12:39:49 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 12:39:49 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 12:41:17 dc2oss09 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394041277 last 1394041277 Mar 5 12:41:17 dc2oss09 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 12:41:25 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:41:25 dc2oss10 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 12:41:28 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1394041288 last 1394041288 Mar 5 12:41:28 dc2oss12 kernel: : Lustre: 16104:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 12:41:44 dc2oss13 kernel: : Lustre: 14407:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c113b3400 cur 1394041304 last 1394041304 Mar 5 12:41:44 dc2oss13 kernel: : Lustre: 14407:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 12:41:53 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) Mar 5 12:41:53 dc2oss05 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 12:41:55 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 179552, left: 179552, last alive is 44 secs ago Mar 5 12:41:55 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 179552, left: 179552, last alive is 44 secs ago Mar 5 12:41:56 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 12:41:56 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:41:58 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805559800 cur 1394041318 last 1394041318 Mar 5 12:41:58 dc2oss15 kernel: : Lustre: 15667:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 12:42:04 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394041324 last 1394041324 Mar 5 12:42:04 dc2oss04 kernel: : Lustre: 16501:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 12:42:33 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818270ce000 cur 1394041353 last 1394041353 Mar 5 12:42:33 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 12:43:03 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 12:43:03 dc2oss02 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:42:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 12:42:43 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 12:43:18 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001a_UUID' is not available for connect (no target) Mar 5 12:43:18 dc2oss04 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 12:43:31 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0085_UUID' is not available for connect (no target) Mar 5 12:43:31 dc2oss14 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 12:43:33 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 12:43:33 dc2oss13 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:43:43 149.165.238.1 kernel: kernel: [4315795.728029] Lustre: 43815:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669105967760 sent from dc2-OST0071-osc-ffff880956851000 to NID 10.10.0.11@o2ib 8s ago has timed out (8s prior to deadline). Mar 5 12:43:43 149.165.238.1 kernel: kernel: [4315795.728042] Lustre: dc2-OST0071-osc-ffff880956851000: Connection to service dc2-OST0071 via nid 10.10.0.11@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 12:43:43 149.165.238.1 kernel: kernel: [4315795.728525] Lustre: dc2-OST0071-osc-ffff880956851000: Connection restored to service dc2-OST0071 using nid 10.10.0.11@o2ib. Mar 5 12:43:43 149.165.238.1 kernel: kernel: [4315795.728546] Lustre: Server dc2-OST0071_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 12:43:39 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1df33800 cur 1394041419 last 1394041419 Mar 5 12:43:39 dc2oss06 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 12:43:44 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0095_UUID' is not available for connect (no target) Mar 5 12:43:44 dc2oss16 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 12:43:51 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a2: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31135835 exp ffff88182acaf400 cur 1394041431 last 1394041431 Mar 5 12:43:51 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 12:44:29 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:44:29 dc2oss16 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 12:44:53 dc2oss03 kernel: : Lustre: dc2-OST0017: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:44:53 dc2oss03 kernel: : Lustre: Skipped 83 previous similar messages Mar 5 12:44:53 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c09da5400 cur 1394041493 last 1394041493 Mar 5 12:44:53 dc2oss03 kernel: : Lustre: 16689:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 82 previous similar messages Mar 5 12:44:33 dc2oss04 kernel: : Lustre: dc2-OST0028: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:44:33 dc2oss04 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:44:57 dc2oss02 kernel: : Lustre: dc2-OST000b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:44:57 dc2oss02 kernel: : Lustre: Skipped 78 previous similar messages Mar 5 12:44:57 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181cfcd800 cur 1394041497 last 1394041497 Mar 5 12:44:57 dc2oss02 kernel: : Lustre: 16397:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 78 previous similar messages Mar 5 12:44:40 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 12:44:40 dc2oss06 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 12:44:35 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000f_UUID' is not available for connect (no target) Mar 5 12:44:35 dc2oss01 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:44:45 dc2oss05 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394041485 last 1394041485 Mar 5 12:44:45 dc2oss05 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 12:44:53 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0066: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c25d96000 cur 1394041493 last 1394041493 Mar 5 12:44:53 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 12:44:44 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:44:44 dc2mds01 kernel: : Lustre: Skipped 30 previous similar messages Mar 5 12:44:44 dc2mds01 kernel: : Lustre: 11195:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394041484 last 1394041484 Mar 5 12:44:44 dc2mds01 kernel: : Lustre: 11195:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages Mar 5 12:45:33 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0029_UUID' is not available for connect (no target) Mar 5 12:45:33 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:45:43 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006c_UUID' is not available for connect (no target) Mar 5 12:45:43 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 12:45:43 dc2oss11 kernel: : Lustre: 15438:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f9ef5400 cur 1394041543 last 1394041543 Mar 5 12:45:43 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t39019319 exp ffff8817fa2cf800 cur 1394041543 last 1394041543 Mar 5 12:45:43 dc2oss11 kernel: : Lustre: 15394:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 12:45:46 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:45:46 dc2oss14 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 12:45:54 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 761952, left: 761952, last alive is 44 secs ago Mar 5 12:45:54 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 761952, left: 761952, last alive is 44 secs ago Mar 5 12:46:03 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:03 dc2oss11 kernel: : Lustre: Skipped 94 previous similar messages Mar 5 12:46:12 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:12 dc2oss13 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 12:46:17 dc2oss08 kernel: : Lustre: dc2-OST0050: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:17 dc2oss08 kernel: : Lustre: Skipped 47 previous similar messages Mar 5 12:46:17 dc2oss08 kernel: : Lustre: 16068:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394041577 last 1394041577 Mar 5 12:46:17 dc2oss08 kernel: : Lustre: 16068:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 46 previous similar messages Mar 5 12:46:12 dc2oss01 kernel: : Lustre: dc2-OST0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:12 dc2oss01 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 12:46:12 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c003ae000 cur 1394041572 last 1394041572 Mar 5 12:46:12 dc2oss01 kernel: : Lustre: 15350:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 12:46:31 dc2oss09 kernel: : Lustre: dc2-OST0057: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:31 dc2oss09 kernel: : Lustre: Skipped 126 previous similar messages Mar 5 12:46:43 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:46:43 dc2oss15 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 12:47:44 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0055_UUID' is not available for connect (no target) Mar 5 12:47:44 dc2oss10 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 12:48:26 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 833632, left: 833632, last alive is 0 secs ago Mar 5 12:48:26 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 833632, left: 833632, last alive is 0 secs ago Mar 5 12:48:31 dc2oss06 kernel: : Lustre: dc2-OST003a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:48:31 dc2oss06 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 12:48:24 dc2oss11 kernel: : Lustre: Service thread pid 25440 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 12:48:24 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 12:48:24 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394041704.25440 Mar 5 12:48:37 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:48:37 dc2oss12 kernel: : Lustre: Skipped 98 previous similar messages Mar 5 12:48:46 dc2oss05 kernel: : Lustre: dc2-OST0034: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:48:46 dc2oss05 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 12:48:48 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 12:48:48 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 12:48:54 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 12:48:54 dc2oss11 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 12:49:17 dc2oss07 kernel: : Lustre: dc2-OST0043: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:49:17 dc2oss07 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 12:49:17 dc2oss07 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0043: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818067fd000 cur 1394041757 last 1394041757 Mar 5 12:49:17 dc2oss07 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 12:49:37 dc2mds01 kernel: : Lustre: 1380:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb03:0x25:0x0]] [valid 68719476736] Mar 5 12:49:37 dc2mds01 kernel: : Lustre: 1380:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 45 previous similar messages Mar 5 12:49:49 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0043_UUID' is not available for connect (no target) Mar 5 12:49:49 dc2oss08 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:50:04 dc2oss11 kernel: : Lustre: 25363:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:50:09 dc2oss11 kernel: : LustreError: 14623:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff8817fa5af850 x1442912236993435/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041809 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:50:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:50:09 dc2oss11 kernel: : Lustre: Skipped 5 previous similar messages Mar 5 12:50:11 dc2oss11 kernel: : Lustre: 14622:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 12:50:16 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 7 active RPCs Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 25448:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880c0effe000 x1442912237001209/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394042561 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 25448:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 12:50:16 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 25453:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bee6a6800 x1442912236993892/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041829 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 25355:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff8817fab0a050 x1442912236993661/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041816 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:50:16 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:50:16 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 14680:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd5a3a400 x1442912236995641/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394042020 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 12:50:16 dc2oss11 kernel: : LustreError: 14680:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 12:50:16 dc2oss11 kernel: : Lustre: Service thread pid 25440 completed after 312.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 12:50:58 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 0 secs ago Mar 5 12:50:58 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 511072, left: 511072, last alive is 0 secs ago Mar 5 12:50:59 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0053_UUID' is not available for connect (no target) Mar 5 12:50:59 dc2oss07 kernel: : LustreError: Skipped 10 previous similar messages Mar 5 12:51:19 dc2oss09 kernel: : Lustre: 16136:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881805060c00 cur 1394041879 last 1394041879 Mar 5 12:51:19 dc2oss09 kernel: : Lustre: 16136:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 12:51:25 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:51:25 dc2oss10 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 12:51:33 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2553a400 cur 1394041893 last 1394041893 Mar 5 12:51:33 dc2oss12 kernel: : Lustre: 15267:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 12:51:46 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0083: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e9a3000 cur 1394041906 last 1394041906 Mar 5 12:51:46 dc2oss13 kernel: : Lustre: 15786:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 12:52:00 dc2oss15 kernel: : Lustre: 15885:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35603425 exp ffff8817c7425400 cur 1394041920 last 1394041920 Mar 5 12:52:00 dc2oss15 kernel: : Lustre: 15885:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 12:52:08 dc2oss11 kernel: : LustreError: 25436:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 100+0s req@ffff880bd4f23c00 x1442912237001440/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394041928 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 12:52:08 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 12:52:08 dc2oss11 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 12:52:08 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394041928 last 1394041928 Mar 5 12:52:08 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 68 previous similar messages Mar 5 12:52:32 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 12:52:32 dc2oss15 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 12:52:35 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394041955 last 1394041955 Mar 5 12:52:35 dc2oss14 kernel: : Lustre: 14561:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 12:53:21 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0037_UUID' is not available for connect (no target) Mar 5 12:53:21 dc2oss05 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 12:53:23 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001b_UUID' is not available for connect (no target) Mar 5 12:53:23 dc2oss04 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:53:30 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 36544, left: 36544, last alive is 24 secs ago Mar 5 12:53:30 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 36544, left: 36544, last alive is 24 secs ago Mar 5 12:53:35 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0089_UUID' is not available for connect (no target) Mar 5 12:53:35 dc2oss13 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 12:53:58 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0009_UUID' is not available for connect (no target) Mar 5 12:53:58 dc2oss02 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 12:53:39 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0082_UUID' is not available for connect (no target) Mar 5 12:53:39 dc2oss14 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 12:53:43 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394042023 last 1394042023 Mar 5 12:53:43 dc2oss06 kernel: : Lustre: 16036:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 12:53:49 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0094_UUID' is not available for connect (no target) Mar 5 12:53:49 dc2oss16 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 12:53:55 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c037a2000 cur 1394042035 last 1394042035 Mar 5 12:53:55 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 12:54:07 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 279520, left: 279520, last alive is 35 secs ago Mar 5 12:54:07 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 279520, left: 279520, last alive is 35 secs ago Mar 5 12:54:54 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:54:54 dc2oss03 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 12:54:54 dc2oss03 kernel: : Lustre: 17080:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394042094 last 1394042094 Mar 5 12:54:54 dc2oss03 kernel: : Lustre: 17080:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 12:54:34 dc2oss04 kernel: : Lustre: dc2-OST0020: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:54:34 dc2oss04 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 12:54:46 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c07198800 cur 1394042086 last 1394042086 Mar 5 12:54:46 dc2oss05 kernel: : Lustre: 15887:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 12:55:00 dc2oss16 kernel: : Lustre: dc2-OST00a3: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:55:00 dc2oss16 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 12:55:10 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:55:10 dc2oss02 kernel: : Lustre: Skipped 68 previous similar messages Mar 5 12:55:10 dc2oss02 kernel: : Lustre: 16403:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394042110 last 1394042110 Mar 5 12:55:10 dc2oss02 kernel: : Lustre: 16403:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 68 previous similar messages Mar 5 12:55:02 dc2oss10 kernel: : Lustre: 16518:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfc6e0c00 cur 1394042102 last 1394042102 Mar 5 12:55:02 dc2oss10 kernel: : Lustre: 16518:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 12:55:05 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0030_UUID' is not available for connect (no target) Mar 5 12:55:05 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 12:54:58 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 12:54:58 dc2oss01 kernel: : LustreError: Skipped 9 previous similar messages Mar 5 12:55:41 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0021_UUID' is not available for connect (no target) Mar 5 12:55:41 dc2oss03 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 12:55:47 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 12:55:47 dc2oss12 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 12:55:47 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:55:47 dc2oss14 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 12:55:59 dc2oss11 kernel: : Lustre: 15357:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0072: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0eeaf000 cur 1394042159 last 1394042159 Mar 5 12:55:59 dc2oss11 kernel: : Lustre: 15357:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 12:56:03 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:03 dc2mds01 kernel: : Lustre: Skipped 19 previous similar messages Mar 5 12:56:03 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394042163 last 1394042163 Mar 5 12:56:03 dc2mds01 kernel: : Lustre: 9749:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 19 previous similar messages Mar 5 12:56:03 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 12:56:03 dc2mds01 kernel: : LustreError: 9749:0:(obd_class.h:503:obd_set_info_async()) Skipped 20 previous similar messages Mar 5 12:56:26 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 726112, left: 726112, last alive is 5 secs ago Mar 5 12:56:26 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 726112, left: 726112, last alive is 5 secs ago Mar 5 12:56:14 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:14 dc2oss01 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 12:56:14 dc2oss01 kernel: : Lustre: 15526:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394042174 last 1394042174 Mar 5 12:56:14 dc2oss01 kernel: : Lustre: 15526:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 101 previous similar messages Mar 5 12:56:28 dc2oss13 kernel: : Lustre: dc2-OST0086: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:28 dc2oss13 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 12:56:43 dc2oss09 kernel: : Lustre: dc2-OST005c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:43 dc2oss09 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 12:56:44 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:44 dc2oss15 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 12:56:48 dc2oss11 kernel: : Lustre: dc2-OST006a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:56:48 dc2oss11 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 12:57:17 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 403552, left: 403552, last alive is 0 secs ago Mar 5 12:57:17 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 403552, left: 403552, last alive is 0 secs ago Mar 5 12:57:17 dc2oss08 kernel: : Lustre: 14516:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394042230/real 1394042230] req@ffff88138cbc4800 x1456766190358796/t0(0) o104->dc2-OST004e@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394042237 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 12:57:17 dc2oss08 kernel: : LustreError: 138-a: dc2-OST004e: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 12:57:19 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from 00795451-8f38-1361-7a6d-ab2d6c88925e@149.165.226.206@tcp t160490814 exp (null) cur 1394042239 last 0 Mar 5 12:57:19 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 69 previous similar messages Mar 5 12:57:57 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:57:57 dc2oss08 kernel: : Lustre: Skipped 69 previous similar messages Mar 5 12:58:27 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005c_UUID' is not available for connect (no target) Mar 5 12:58:27 dc2oss10 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 12:58:38 dc2oss06 kernel: : Lustre: dc2-OST003c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:58:38 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 12:58:55 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:58:55 dc2oss12 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 12:59:10 dc2oss05 kernel: : Lustre: dc2-OST0033: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:59:10 dc2oss05 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 12:59:17 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:59:17 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 12:59:17 dc2oss07 kernel: : Lustre: Skipped 63 previous similar messages Mar 5 12:59:17 dc2oss07 kernel: : Lustre: Skipped 63 previous similar messages Mar 5 12:59:17 dc2oss07 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394042357 last 1394042357 Mar 5 12:59:17 dc2oss07 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 64 previous similar messages Mar 5 12:59:23 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0076_UUID' is not available for connect (no target) Mar 5 12:59:23 dc2oss11 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 12:59:37 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0065_UUID' is not available for connect (no target) Mar 5 12:59:37 dc2oss09 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 12:59:55 dc2mds01 kernel: : Lustre: 11160:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f796:0x5f74:0x0]] [valid 68719476736] Mar 5 12:59:55 dc2mds01 kernel: : Lustre: 11160:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 50 previous similar messages Mar 5 13:01:05 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 13:01:05 dc2oss03 kernel: : LustreError: 16809:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800b536f800 x1461679860268470/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 0 to 0 dl 1394042481 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:01:05 dc2oss03 kernel: : LustreError: 16809:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:01:05 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:05 dc2oss03 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:01:06 dc2oss03 kernel: : LustreError: 16740:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880883a49c00 x1461679860268471/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 0 to 0 dl 1394042482 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:01:06 dc2oss03 kernel: : LustreError: 16740:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:01:06 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:06 dc2oss03 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:01:06 dc2oss03 kernel: : LustreError: 16800:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806b934ac00 x1461679860268446/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 0 to 0 dl 1394042478 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:01:06 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:15 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0046_UUID' is not available for connect (no target) Mar 5 13:01:15 dc2oss08 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:01:19 dc2oss09 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf532ac00 cur 1394042479 last 1394042479 Mar 5 13:01:19 dc2oss09 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 127 previous similar messages Mar 5 13:01:28 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:01:28 dc2oss10 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 13:01:30 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 13:01:30 dc2oss08 kernel: : LustreError: 32178:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88048ce54400 x1461679860268442/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042549 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:01:30 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:30 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:01:31 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:31 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:01:31 dc2oss08 kernel: : LustreError: 27808:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd40d3400 x1461679860268473/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042553 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:01:31 dc2oss08 kernel: : LustreError: 27808:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:01:31 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:01:31 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:01:36 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394042496 last 1394042496 Mar 5 13:01:36 dc2oss12 kernel: : Lustre: 16146:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 136 previous similar messages Mar 5 13:01:44 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004c_UUID' is not available for connect (no target) Mar 5 13:01:44 dc2oss07 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 13:01:46 dc2oss13 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from 86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp t0 exp ffff881723701000 cur 1394042506 last 1394042506 Mar 5 13:01:46 dc2oss13 kernel: : Lustre: 16054:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 141 previous similar messages Mar 5 13:02:04 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394042524 last 1394042524 Mar 5 13:02:04 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 134 previous similar messages Mar 5 13:02:18 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180672d800 cur 1394042538 last 1394042538 Mar 5 13:02:18 dc2oss04 kernel: : Lustre: 16042:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 144 previous similar messages Mar 5 13:02:40 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394042560 last 1394042560 Mar 5 13:02:40 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 13:03:01 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 13:03:02 dc2oss03 kernel: : LustreError: 21015:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880239682c00 x1461679860270014/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042657 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:03:02 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:03:06 dc2oss11 kernel: : Lustre: 14703:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 13:03:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 7 active RPCs Mar 5 13:03:09 dc2oss11 kernel: : LustreError: 8748:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd5f18400 x1442912237004892/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394042940 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:03:09 dc2oss11 kernel: : LustreError: 8748:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:03:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:03:09 dc2oss11 kernel: : LustreError: 8789:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8817fa5b6050 x1442912237004433/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394042884 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:03:09 dc2oss11 kernel: : LustreError: 8789:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 13:03:09 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:03:09 dc2oss11 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:03:56 dc2oss02 kernel: : Lustre: dc2-OST0011: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:03:57 dc2oss02 kernel: : LustreError: 20288:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf738c850 x1461679899140347/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042681 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:03:57 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:03:45 dc2oss14 kernel: : Lustre: dc2-OST008c: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:03:46 dc2oss14 kernel: : LustreError: 22741:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc6c9d800 x1461679899140648/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042686 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:03:46 dc2oss14 kernel: : LustreError: 22741:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:03:46 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:03:49 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 13:03:49 dc2oss14 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 13:04:03 dc2oss02 kernel: : Lustre: dc2-OST000e: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 13:04:03 dc2oss02 kernel: : LustreError: 16029:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88078dcc4000 x1461679899140339/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042680 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:04:03 dc2oss02 kernel: : LustreError: 16029:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:04:03 dc2oss02 kernel: : Lustre: dc2-OST000e: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:04:03 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:03:56 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:03:56 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from c1ce5d4a-1b3a-c884-4cf4-e623e681091b@149.165.235.51@tcp t0 exp ffff881072f17000 cur 1394042636 last 1394042636 Mar 5 13:03:56 dc2oss16 kernel: : Lustre: 17155:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 13:04:10 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f26fc00 cur 1394042650 last 1394042650 Mar 5 13:04:10 dc2oss06 kernel: : Lustre: 15964:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 13:04:15 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 13:04:15 dc2oss16 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:04:27 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0001_UUID' is not available for connect (no target) Mar 5 13:04:27 dc2oss02 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 13:04:16 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008d_UUID' is not available for connect (no target) Mar 5 13:04:16 dc2oss13 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:04:29 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 13:04:29 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 13:04:35 dc2oss05 kernel: : Lustre: dc2-OST002d: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:04:35 dc2oss05 kernel: : LustreError: 25201:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88025117a000 x1461679899140359/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042757 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:04:35 dc2oss05 kernel: : LustreError: 25201:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:04:35 dc2oss05 kernel: : Lustre: dc2-OST002d: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:04:35 dc2oss05 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:04:26 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 13:04:26 dc2oss04 kernel: : LustreError: 20209:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880725e5f800 x1461679899140344/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042747 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:04:26 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:04:27 dc2oss04 kernel: : LustreError: 19750:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b07dbf800 x1461679899140353/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042748 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:04:27 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:04:36 dc2oss04 kernel: : Lustre: dc2-OST0025: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:04:36 dc2oss04 kernel: : Lustre: Skipped 159 previous similar messages Mar 5 13:05:04 dc2oss03 kernel: : Lustre: dc2-OST0017: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 13:05:04 dc2oss03 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 13:05:04 dc2oss03 kernel: : Lustre: 16936:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0017: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff881037a13800 cur 1394042704 last 1394042704 Mar 5 13:05:04 dc2oss03 kernel: : Lustre: 16936:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 13:04:48 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff880ca7ffdc00 cur 1394042688 last 1394042688 Mar 5 13:04:48 dc2oss05 kernel: : Lustre: 16040:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 188 previous similar messages Mar 5 13:05:00 dc2oss16 kernel: : Lustre: dc2-OST009f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:05:00 dc2oss16 kernel: : Lustre: Skipped 116 previous similar messages Mar 5 13:05:13 dc2oss02 kernel: : Lustre: dc2-OST000b: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) reconnecting Mar 5 13:05:13 dc2oss02 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 13:05:13 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff8811f81abc00 cur 1394042713 last 1394042713 Mar 5 13:05:13 dc2oss02 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 129 previous similar messages Mar 5 13:05:02 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from 48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp t0 exp ffff881181e72400 cur 1394042702 last 1394042702 Mar 5 13:05:02 dc2oss10 kernel: : Lustre: 15880:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 125 previous similar messages Mar 5 13:05:07 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009f_UUID' is not available for connect (no target) Mar 5 13:05:07 dc2oss15 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:05:07 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 13:05:07 dc2oss06 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:05:04 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0011_UUID' is not available for connect (no target) Mar 5 13:05:04 dc2oss01 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 13:05:24 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 13:05:24 dc2oss08 kernel: : LustreError: 32236:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8818038bf050 x1461679899140363/t0(0) o4->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/416 e 1 to 0 dl 1394042849 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:05:24 dc2oss08 kernel: : LustreError: 32236:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:05:24 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO write error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:05:24 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:05:24 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO write error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:05:48 dc2oss14 kernel: : Lustre: dc2-OST0090: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:05:48 dc2oss14 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 13:06:07 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001d_UUID' is not available for connect (no target) Mar 5 13:06:07 dc2oss04 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 13:06:18 dc2oss02 kernel: : Lustre: dc2-OST0011: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:06:18 dc2oss02 kernel: : Lustre: dc2-OST000e: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 13:06:18 dc2oss02 kernel: : LustreError: 16060:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880791da3c00 x1461679899141947/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394042845 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:06:18 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:06:14 dc2oss11 kernel: : Lustre: 15542:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394042774 last 1394042774 Mar 5 13:06:14 dc2oss11 kernel: : Lustre: 15542:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 264 previous similar messages Mar 5 13:06:21 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006d_UUID' is not available for connect (no target) Mar 5 13:06:21 dc2oss12 kernel: : LustreError: Skipped 15 previous similar messages Mar 5 13:06:24 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:06:24 dc2oss01 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 13:06:24 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394042784 last 1394042784 Mar 5 13:06:24 dc2oss01 kernel: : Lustre: 15390:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 13:06:28 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:06:28 dc2oss13 kernel: : Lustre: Skipped 142 previous similar messages Mar 5 13:06:28 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 13:06:28 dc2mds01 kernel: : Lustre: Skipped 32 previous similar messages Mar 5 13:06:28 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394042788 last 1394042788 Mar 5 13:06:28 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 34 previous similar messages Mar 5 13:06:28 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:06:28 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 23 previous similar messages Mar 5 13:06:49 dc2oss09 kernel: : Lustre: dc2-OST005e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:06:49 dc2oss09 kernel: : Lustre: Skipped 124 previous similar messages Mar 5 13:07:04 dc2oss11 kernel: : Lustre: dc2-OST0071: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:07:04 dc2oss11 kernel: : Lustre: Skipped 257 previous similar messages Mar 5 13:07:09 dc2oss15 kernel: : Lustre: dc2-OST0093: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:07:09 dc2oss15 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 13:07:31 dc2oss08 kernel: : Lustre: 15858:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394042851 last 1394042851 Mar 5 13:07:31 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881803153400 cur 1394042851 last 1394042851 Mar 5 13:07:31 dc2oss08 kernel: : Lustre: 15913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 13:08:02 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 13:08:02 dc2oss03 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 13:07:59 dc2oss08 kernel: : Lustre: dc2-OST004e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:07:59 dc2oss08 kernel: : Lustre: Skipped 129 previous similar messages Mar 5 13:08:31 dc2oss08 kernel: : Lustre: 6624:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394042904/real 0] req@ffff880c03504c00 x1456766190360172/t0(0) o105->dc2-OST0053@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394042911 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:08:31 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0053: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:08:41 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 13:08:41 dc2oss10 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:08:44 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:08:44 dc2oss06 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 13:09:02 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:09:02 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:09:02 dc2oss12 kernel: : Lustre: Skipped 132 previous similar messages Mar 5 13:09:02 dc2oss12 kernel: : Lustre: Skipped 132 previous similar messages Mar 5 13:09:13 dc2oss05 kernel: : Lustre: dc2-OST002d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:09:13 dc2oss05 kernel: : Lustre: Skipped 175 previous similar messages Mar 5 13:09:22 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:09:22 dc2oss07 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:09:22 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394042962 last 1394042962 Mar 5 13:09:22 dc2oss07 kernel: : Lustre: 15892:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 13:09:49 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 13:09:49 dc2oss09 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 13:09:56 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0077_UUID' is not available for connect (no target) Mar 5 13:09:56 dc2oss11 kernel: : LustreError: Skipped 45 previous similar messages Mar 5 13:09:59 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb0c:0x27:0x0]] [valid 68719476736] Mar 5 13:09:59 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 53 previous similar messages Mar 5 13:11:24 dc2oss09 kernel: : Lustre: 16302:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394043084 last 1394043084 Mar 5 13:11:24 dc2oss09 kernel: : Lustre: 16302:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 13:11:33 dc2oss10 kernel: : Lustre: dc2-OST0065: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:11:33 dc2oss10 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 13:11:37 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394043097 last 1394043097 Mar 5 13:11:37 dc2oss12 kernel: : Lustre: 15295:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 13:11:58 dc2oss13 kernel: : Lustre: 16010:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818047d7000 cur 1394043118 last 1394043118 Mar 5 13:11:58 dc2oss13 kernel: : Lustre: 16010:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 13:11:49 dc2oss04 kernel: : Lustre: 23362:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394043102/real 1394043102] req@ffff880b843a3000 x1461670963956058/t0(0) o105->dc2-OST0025@149.165.229.21@tcp:15/16 lens 344/192 e 0 to 1 dl 1394043109 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:11:49 dc2oss04 kernel: : LustreError: 138-a: dc2-OST0025: A client on nid 149.165.229.21@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:12:00 dc2oss08 kernel: : Lustre: 7147:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394043112/real 0] req@ffff880b73146800 x1456766190360760/t0(0) o105->dc2-OST004c@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394043120 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:12:00 dc2oss08 kernel: : LustreError: 138-a: dc2-OST004c: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:12:05 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394043125 last 1394043125 Mar 5 13:12:05 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 112 previous similar messages Mar 5 13:12:23 dc2oss04 kernel: : Lustre: 16359:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bffed2000 cur 1394043143 last 1394043143 Mar 5 13:12:23 dc2oss04 kernel: : Lustre: 16359:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 13:12:29 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0050_UUID' is not available for connect (no target) Mar 5 13:12:29 dc2oss07 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:12:44 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394043164 last 1394043164 Mar 5 13:12:44 dc2oss14 kernel: : Lustre: 15763:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 13:12:45 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 13:12:45 dc2oss08 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 13:13:39 149.165.238.1 kernel: kernel: [4317591.596389] LustreError: 12276:0:(file.c:116:ll_close_inode_openhandle()) inode 144943094477911456 mdc close failed: rc = -116 Mar 5 13:13:51 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007e_UUID' is not available for connect (no target) Mar 5 13:13:51 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:13:58 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394043238 last 1394043238 Mar 5 13:13:58 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 123 previous similar messages Mar 5 13:14:16 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 13:14:16 dc2oss16 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:14:22 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394043262 last 1394043262 Mar 5 13:14:22 dc2oss06 kernel: : Lustre: 16008:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 13:15:10 dc2oss03 kernel: : Lustre: dc2-OST001d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:15:10 dc2oss03 kernel: : Lustre: Skipped 80 previous similar messages Mar 5 13:15:10 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c024e0000 cur 1394043310 last 1394043310 Mar 5 13:15:10 dc2oss03 kernel: : Lustre: 17439:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 79 previous similar messages Mar 5 13:14:46 dc2oss04 kernel: : Lustre: dc2-OST0023: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:14:46 dc2oss04 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 13:14:54 dc2oss05 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0034: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be66fcc00 cur 1394043294 last 1394043294 Mar 5 13:14:54 dc2oss05 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 107 previous similar messages Mar 5 13:15:04 dc2oss10 kernel: : Lustre: 16566:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11503000 cur 1394043304 last 1394043304 Mar 5 13:15:04 dc2oss10 kernel: : Lustre: 16566:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 13:15:06 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0036_UUID' is not available for connect (no target) Mar 5 13:15:06 dc2oss05 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 13:15:19 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:15:19 dc2oss02 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:15:19 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394043319 last 1394043318 Mar 5 13:15:19 dc2oss02 kernel: : Lustre: 16399:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 13:15:31 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 13:15:31 dc2oss02 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 13:15:15 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 13:15:15 dc2oss01 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:15:39 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 13:15:39 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 13:15:49 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:15:49 dc2oss16 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 13:15:49 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:15:49 dc2oss14 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 13:15:56 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 13:15:56 dc2oss15 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:16:02 dc2oss08 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 113s: evicting client at 149.165.226.206@tcp ns: filter-dc2-OST0051_UUID lock: ffff88166d9eb000/0x9b00870506d9506 lrc: 3/0,0 mode: PR/PR res: 23603679/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfd05479c5918d94f expref: 634 pid: 15999 timeout 9058597909 Mar 5 13:16:16 dc2oss11 kernel: : Lustre: 16346:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826488800 cur 1394043376 last 1394043376 Mar 5 13:16:16 dc2oss11 kernel: : Lustre: 16346:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 138 previous similar messages Mar 5 13:16:28 dc2oss13 kernel: : Lustre: dc2-OST0083: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:16:28 dc2oss13 kernel: : Lustre: Skipped 128 previous similar messages Mar 5 13:16:28 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:16:28 dc2mds01 kernel: : Lustre: Skipped 25 previous similar messages Mar 5 13:16:28 dc2mds01 kernel: : Lustre: 14551:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394043388 last 1394043388 Mar 5 13:16:28 dc2mds01 kernel: : Lustre: 14551:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 25 previous similar messages Mar 5 13:16:34 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:16:34 dc2mds01 kernel: : LustreError: 9750:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 13:16:32 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:16:32 dc2oss01 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 13:16:32 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394043392 last 1394043392 Mar 5 13:16:32 dc2oss01 kernel: : Lustre: 14991:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 90 previous similar messages Mar 5 13:16:50 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 13:16:50 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 13:16:51 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 198176, left: 198176, last alive is 24 secs ago Mar 5 13:16:51 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 198176, left: 198176, last alive is 24 secs ago Mar 5 13:16:53 dc2oss09 kernel: : Lustre: dc2-OST0058: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:16:53 dc2oss09 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 13:17:09 dc2oss15 kernel: : Lustre: dc2-OST0096: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:17:09 dc2oss15 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 13:17:31 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04ae7400 cur 1394043451 last 1394043451 Mar 5 13:17:31 dc2oss08 kernel: : Lustre: 15972:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 13:17:56 dc2oss11 kernel: : Lustre: dc2-OST006e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:17:56 dc2oss11 kernel: : Lustre: dc2-OST0070: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:17:56 dc2oss11 kernel: : Lustre: Skipped 140 previous similar messages Mar 5 13:17:56 dc2oss11 kernel: : Lustre: Skipped 140 previous similar messages Mar 5 13:17:48 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 13:17:48 dc2oss04 kernel: : LustreError: Skipped 13 previous similar messages Mar 5 13:17:54 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 112800, left: 112800, last alive is 23 secs ago Mar 5 13:17:54 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 112800, left: 112800, last alive is 23 secs ago Mar 5 13:18:04 dc2oss08 kernel: : Lustre: dc2-OST004a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:18:04 dc2oss08 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 13:18:06 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 13:18:06 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:18:29 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 13:18:29 dc2oss03 kernel: : LustreError: 20387:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880767c28c00 x1461679860288976/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043551 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:18:29 dc2oss03 kernel: : LustreError: 20387:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:18:29 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:18:29 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:18:29 dc2oss03 kernel: : LustreError: 20923:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880883a49000 x1461679860288966/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043548 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:18:29 dc2oss03 kernel: : LustreError: 20923:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:18:29 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:18:29 dc2oss03 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:18:16 dc2oss05 kernel: : Lustre: 15852:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394043489/real 1394043489] req@ffff88181a7a8800 x1456766146495206/t0(0) o104->dc2-OST002c@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394043496 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:18:16 dc2oss05 kernel: : Lustre: 15852:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Mar 5 13:18:16 dc2oss05 kernel: : LustreError: 138-a: dc2-OST002c: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 13:18:26 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 4 active RPCs Mar 5 13:18:26 dc2oss08 kernel: : LustreError: 27762:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd5827400 x1461679860288975/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043551 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:18:26 dc2oss08 kernel: : LustreError: 27762:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 13:18:26 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:18:26 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:18:27 dc2oss08 kernel: : LustreError: 14649:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8818142f0850 x1461679860288969/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043549 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:18:27 dc2oss08 kernel: : LustreError: 14649:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:18:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd_cb.c:2520:ksocknal_check_peer_timeouts()) Total 2 stale ZC_REQs for peer 149.165.229.66@tcp detected; the oldest(ffff880ffe0e0000) timed out 1 secs ago, resid: 0, wmem: 673280 Mar 5 13:18:23 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd_cb.c:2520:ksocknal_check_peer_timeouts()) Total 2 stale ZC_REQs for peer 149.165.229.66@tcp detected; the oldest(ffff880ffe0e0000) timed out 1 secs ago, resid: 0, wmem: 673280 Mar 5 13:18:23 dc2oss05 kernel: : LustreError: 18558:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.226.206@tcp arrived at 1394043503 with bad export cookie 17883493847239786182 Mar 5 13:18:51 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 13:18:51 dc2oss03 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 13:18:49 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:18:49 dc2oss06 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 13:18:52 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 13:18:52 dc2oss10 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:19:11 dc2oss14 kernel: : Lustre: dc2-OST008c: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 13:19:15 dc2oss05 kernel: : Lustre: dc2-OST002f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:19:15 dc2oss05 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:19:08 dc2oss12 kernel: : Lustre: dc2-OST007d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:19:08 dc2oss12 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:19:12 dc2oss14 kernel: : LustreError: 22829:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880056452400 x1461679899163277/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043589 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:19:12 dc2oss14 kernel: : LustreError: 22829:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:19:12 dc2oss14 kernel: : Lustre: dc2-OST008c: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:19:14 dc2oss11 kernel: : Lustre: Service thread pid 8805 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:19:14 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:19:14 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043554.8805 Mar 5 13:19:17 dc2oss11 kernel: : Lustre: Service thread pid 25427 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:19:17 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:19:17 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043557.25427 Mar 5 13:19:22 dc2oss11 kernel: : Lustre: Service thread pid 25380 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:19:22 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:19:22 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043562.25380 Mar 5 13:19:22 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 367712, left: 367712, last alive is 25 secs ago Mar 5 13:19:22 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 367712, left: 367712, last alive is 25 secs ago Mar 5 13:20:19 dc2oss03 kernel: : Lustre: dc2-OST001e: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 13:20:19 dc2oss03 kernel: : LustreError: 16740:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff881805473850 x1461679860290084/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043703 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:20:19 dc2oss03 kernel: : LustreError: 16740:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:20:19 dc2oss03 kernel: : Lustre: dc2-OST001e: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:20:19 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:20:00 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:20:00 dc2oss07 kernel: : Lustre: Skipped 86 previous similar messages Mar 5 13:20:00 dc2oss07 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c12415000 cur 1394043600 last 1394043600 Mar 5 13:20:00 dc2oss07 kernel: : Lustre: 16105:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 86 previous similar messages Mar 5 13:20:07 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) Mar 5 13:20:07 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 13:20:00 dc2mds01 kernel: : Lustre: 1370:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fb09:0x27:0x0]] [valid 68719476736] Mar 5 13:20:00 dc2mds01 kernel: : Lustre: 1370:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 55 previous similar messages Mar 5 13:20:22 dc2oss11 kernel: : Lustre: Service thread pid 14637 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:20:22 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:20:22 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043622.14637 Mar 5 13:20:22 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:20:22 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043622.25320 Mar 5 13:20:28 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 13:20:28 dc2oss09 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:21:13 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 13:21:14 dc2oss08 kernel: : LustreError: 14627:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806886c9400 x1461679860290096/t0(0) o3->86045972-78e0-90f8-6a5f-e2580eb516f1@149.165.229.66@tcp:0/0 lens 448/400 e 1 to 0 dl 1394043714 ref 1 fl Interpret:/2/0 rc 0/0 Mar 5 13:21:14 dc2oss08 kernel: : LustreError: 14627:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:21:14 dc2oss08 kernel: : Lustre: dc2-OST004c: Bulk IO read error with 86045972-78e0-90f8-6a5f-e2580eb516f1 (at 149.165.229.66@tcp), client will retry: rc -110 Mar 5 13:21:14 dc2oss08 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:21:29 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881808972c00 cur 1394043689 last 1394043689 Mar 5 13:21:29 dc2oss09 kernel: : Lustre: 14533:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 13:21:34 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:21:34 dc2oss10 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 13:21:37 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818198ec000 cur 1394043697 last 1394043697 Mar 5 13:21:37 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:21:58 dc2oss13 kernel: : Lustre: 15998:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181efc0000 cur 1394043718 last 1394043718 Mar 5 13:21:58 dc2oss13 kernel: : Lustre: 15998:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 142 previous similar messages Mar 5 13:22:08 dc2oss15 kernel: : Lustre: 15738:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0097: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf83cf000 cur 1394043728 last 1394043728 Mar 5 13:22:08 dc2oss15 kernel: : Lustre: 15738:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 131 previous similar messages Mar 5 13:22:25 dc2oss04 kernel: : Lustre: 16101:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0024: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01dcf400 cur 1394043745 last 1394043745 Mar 5 13:22:25 dc2oss04 kernel: : Lustre: 16101:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 84 previous similar messages Mar 5 13:22:40 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) Mar 5 13:22:40 dc2oss07 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 13:23:19 dc2oss14 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394043799 last 1394043799 Mar 5 13:23:19 dc2oss14 kernel: : Lustre: 14519:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 13:23:34 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0042_UUID' is not available for connect (no target) Mar 5 13:23:34 dc2oss08 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 13:24:08 dc2oss16 kernel: : Lustre: 17320:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818061b0800 cur 1394043848 last 1394043848 Mar 5 13:24:08 dc2oss16 kernel: : Lustre: 17320:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 109 previous similar messages Mar 5 13:24:25 dc2oss06 kernel: : Lustre: 15870:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0039: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111d0c00 cur 1394043865 last 1394043865 Mar 5 13:24:25 dc2oss06 kernel: : Lustre: 15870:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 13:24:29 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0095_UUID' is not available for connect (no target) Mar 5 13:24:29 dc2oss16 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 13:24:46 dc2oss11 kernel: : Lustre: Service thread pid 14630 was inactive for 374.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:24:46 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:24:46 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:24:46 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394043886.14630 Mar 5 13:24:50 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 13:24:50 dc2oss14 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 13:24:58 dc2oss05 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6605000 cur 1394043898 last 1394043898 Mar 5 13:24:58 dc2oss05 kernel: : Lustre: 15843:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 13:24:55 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:24:55 dc2oss04 kernel: : Lustre: Skipped 85 previous similar messages Mar 5 13:25:13 dc2oss10 kernel: : Lustre: 16579:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394043913 last 1394043913 Mar 5 13:25:13 dc2oss10 kernel: : Lustre: 16579:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 13:25:13 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003d_UUID' is not available for connect (no target) Mar 5 13:25:13 dc2oss05 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:25:24 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:25:24 dc2oss02 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 13:25:24 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394043924 last 1394043924 Mar 5 13:25:24 dc2oss02 kernel: : Lustre: 16161:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 13:25:44 dc2oss03 kernel: : Lustre: dc2-OST0015: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) reconnecting Mar 5 13:25:44 dc2oss03 kernel: : Lustre: Skipped 89 previous similar messages Mar 5 13:25:44 dc2oss03 kernel: : Lustre: 17521:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp t0 exp ffff880eece7a400 cur 1394043944 last 1394043944 Mar 5 13:25:44 dc2oss03 kernel: : Lustre: 17521:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 13:25:18 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST000c_UUID' is not available for connect (no target) Mar 5 13:25:18 dc2oss01 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 13:25:45 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0008_UUID' is not available for connect (no target) Mar 5 13:25:45 dc2oss02 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 13:25:48 dc2oss06 kernel: : Lustre: dc2-OST0039: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:25:48 dc2oss06 kernel: : LustreError: 6891:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8802787ecc00 x1461680520983421/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394043968 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:25:48 dc2oss06 kernel: : Lustre: dc2-OST0039: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:25:49 dc2oss06 kernel: : LustreError: 6846:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8809d58f5c00 x1461680520983404/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394043968 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:25:49 dc2oss06 kernel: : Lustre: dc2-OST0039: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:25:49 dc2oss14 kernel: : Lustre: dc2-OST008c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:25:49 dc2oss14 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 13:25:53 dc2oss16 kernel: : Lustre: dc2-OST00a6: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:25:53 dc2oss16 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 13:25:59 dc2oss12 kernel: : Lustre: dc2-OST007c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 13:25:59 dc2oss12 kernel: : LustreError: 9253:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880c00663850 x1461680520983398/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394043965 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:25:59 dc2oss12 kernel: : LustreError: 9253:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:25:59 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:25:59 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:26:00 dc2oss12 kernel: : LustreError: 9194:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88138b49d050 x1461680520983412/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394043966 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:26:00 dc2oss12 kernel: : LustreError: 9194:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:26:00 dc2oss12 kernel: : Lustre: dc2-OST007c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:26:00 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:26:01 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a6_UUID' is not available for connect (no target) Mar 5 13:26:01 dc2oss15 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 13:26:17 dc2oss11 kernel: : Lustre: 14502:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394043977 last 1394043977 Mar 5 13:26:17 dc2oss11 kernel: : Lustre: 14502:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 124 previous similar messages Mar 5 13:26:18 dc2oss07 kernel: : Lustre: dc2-OST0041: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 8 active RPCs Mar 5 13:26:18 dc2oss07 kernel: : LustreError: 25218:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be5c11400 x1461680520984367/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394044044 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:26:18 dc2oss07 kernel: : LustreError: 25218:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages Mar 5 13:26:18 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:26:18 dc2oss07 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:26:18 dc2oss07 kernel: : LustreError: 6358:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88097e367800 x1461680520984379/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 0 to 0 dl 1394044044 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:26:18 dc2oss07 kernel: : Lustre: dc2-OST0041: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:26:29 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 13:26:29 dc2oss13 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:26:29 dc2oss13 kernel: : Lustre: dc2-OST0081: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:26:29 dc2oss13 kernel: : Lustre: Skipped 119 previous similar messages Mar 5 13:26:31 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 13:26:31 dc2mds01 kernel: : Lustre: Skipped 43 previous similar messages Mar 5 13:26:31 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394043991 last 1394043991 Mar 5 13:26:31 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 43 previous similar messages Mar 5 13:26:38 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:26:38 dc2mds01 kernel: : LustreError: 2352:0:(obd_class.h:503:obd_set_info_async()) Skipped 35 previous similar messages Mar 5 13:26:35 dc2oss01 kernel: : Lustre: dc2-OST0002: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:26:35 dc2oss01 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 13:26:35 dc2oss01 kernel: : Lustre: 15446:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bead7ec00 cur 1394043995 last 1394043995 Mar 5 13:26:35 dc2oss01 kernel: : Lustre: 15446:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 13:26:53 dc2oss09 kernel: : Lustre: dc2-OST0056: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:26:53 dc2oss09 kernel: : Lustre: Skipped 157 previous similar messages Mar 5 13:27:01 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 13:27:01 dc2oss06 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:27:12 dc2oss15 kernel: : Lustre: dc2-OST0094: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:27:12 dc2oss15 kernel: : Lustre: Skipped 137 previous similar messages Mar 5 13:27:22 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 744032, left: 744032, last alive is 27 secs ago Mar 5 13:27:22 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 744032, left: 744032, last alive is 27 secs ago Mar 5 13:27:33 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182ae41000 cur 1394044053 last 1394044053 Mar 5 13:27:33 dc2oss08 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 161 previous similar messages Mar 5 13:27:37 dc2oss11 kernel: : Lustre: 14648:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-103), not sending early reply Mar 5 13:27:40 dc2oss11 kernel: : Lustre: 8788:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-103), not sending early reply Mar 5 13:27:42 dc2oss11 kernel: : LustreError: 8805:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 708+0s req@ffff880bd30e0800 x1442912237017873/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394044062 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:27:42 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:27:42 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:27:42 dc2oss11 kernel: : Lustre: Service thread pid 8805 completed after 708.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:27:45 dc2oss11 kernel: : LustreError: 25427:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 708+0s req@ffff880bc704c400 x1442912237017877/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394044065 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:27:45 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:27:45 dc2oss11 kernel: : Lustre: Service thread pid 25427 completed after 708.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:27:45 dc2oss11 kernel: : Lustre: 14648:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-103), not sending early reply Mar 5 13:28:07 dc2oss03 kernel: : Lustre: dc2-OST001c: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 1 active RPCs Mar 5 13:28:07 dc2oss03 kernel: : LustreError: 21175:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8806b5874400 x1461680520983407/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394044192 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:28:07 dc2oss03 kernel: : LustreError: 21175:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:28:07 dc2oss03 kernel: : Lustre: dc2-OST001c: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:28:07 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:27:50 dc2oss11 kernel: : LustreError: 25380:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 708+0s req@ffff88019fe61400 x1442912237017882/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 3 to 0 dl 1394044070 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:27:50 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:27:50 dc2oss11 kernel: : Lustre: Service thread pid 25380 completed after 708.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:27:56 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:27:56 dc2oss11 kernel: : Lustre: Skipped 145 previous similar messages Mar 5 13:28:08 dc2oss11 kernel: : Lustre: 25380:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-66), not sending early reply Mar 5 13:28:13 dc2oss11 kernel: : LustreError: 14637:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 671+0s req@ffff880bd624ac00 x1442912237018383/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 2 to 0 dl 1394044093 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:28:13 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:28:13 dc2oss11 kernel: : Lustre: Service thread pid 14637 completed after 671.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:28:08 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:28:08 dc2oss08 kernel: : Lustre: Skipped 163 previous similar messages Mar 5 13:28:09 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 13:28:09 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:28:18 dc2oss04 kernel: : Lustre: Service thread pid 20286 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:28:18 dc2oss04 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:18 dc2oss04 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:18 dc2oss04 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:18 dc2oss04 kernel: : LustreError: dumping log to /tmp/lustre-log.1394044098.20286 Mar 5 13:28:13 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0070_UUID' is not available for connect (no target) Mar 5 13:28:13 dc2oss12 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 13:28:17 dc2oss06 kernel: : Lustre: Service thread pid 7076 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:28:17 dc2oss06 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:17 dc2oss06 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:17 dc2oss06 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:17 dc2oss06 kernel: : LustreError: dumping log to /tmp/lustre-log.1394044097.7076 Mar 5 13:28:29 dc2oss11 kernel: : Lustre: 25448:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/3), not sending early reply Mar 5 13:28:29 dc2oss11 kernel: : Lustre: 25448:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Mar 5 13:28:14 dc2oss01 kernel: : Lustre: Service thread pid 15212 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:28:14 dc2oss01 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:28:14 dc2oss01 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:14 dc2oss01 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:14 dc2oss01 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:14 dc2oss01 kernel: : LustreError: dumping log to /tmp/lustre-log.1394044094.15212 Mar 5 13:28:15 dc2oss01 kernel: : Lustre: Service thread pid 19586 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:28:15 dc2oss01 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:15 dc2oss01 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:15 dc2oss01 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:15 dc2oss01 kernel: : LustreError: dumping log to /tmp/lustre-log.1394044095.19586 Mar 5 13:28:24 dc2oss04 kernel: : Lustre: Service thread pid 20469 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : LustreError: dumping log to /tmp/lustre-log.1394044104.20469 Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Mar 5 13:28:24 dc2oss04 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:28:31 dc2oss06 kernel: : Lustre: dc2-OST0039: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 13:28:31 dc2oss06 kernel: : LustreError: 31866:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8805d0796c00 x1461680520984392/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 1 to 0 dl 1394044266 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:28:31 dc2oss06 kernel: : Lustre: dc2-OST0039: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:28:34 dc2oss11 kernel: : LustreError: 14630:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 602+0s req@ffff880bd7359400 x1442912237019539/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 1 to 0 dl 1394044114 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:28:34 dc2oss11 kernel: : LustreError: 14630:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:28:34 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:28:34 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:28:34 dc2oss11 kernel: : Lustre: Service thread pid 14630 completed after 602.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:28:34 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:28:51 dc2oss06 kernel: : Lustre: dc2-OST0039: Client e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp) reconnecting Mar 5 13:28:51 dc2oss06 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 13:29:15 dc2oss05 kernel: : Lustre: dc2-OST002c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:29:15 dc2oss05 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 13:29:09 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:29:09 dc2oss12 kernel: : Lustre: Skipped 100 previous similar messages Mar 5 13:29:22 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 13:29:23 dc2oss11 kernel: : LustreError: 8673:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880783754400 x1442912237025661/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394044898 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:29:23 dc2oss11 kernel: : LustreError: 8673:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:29:23 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:29:23 dc2oss11 kernel: : LustreError: 25356:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880cee7fa050 x1442912237024099/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394044455 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:29:24 dc2xfer03 kernel: : Lustre: 2643:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 13:29:24 dc2xfer03 kernel: : Lustre: 2643:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 13:29:48 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0054_UUID' is not available for connect (no target) Mar 5 13:29:48 dc2oss10 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:30:06 dc2mds01 kernel: : Lustre: 14551:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fafe:0x2f:0x0]] [valid 68719476736] Mar 5 13:30:06 dc2mds01 kernel: : Lustre: 14551:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 40 previous similar messages Mar 5 13:30:28 dc2oss07 kernel: : Lustre: dc2-OST0042: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:30:28 dc2oss07 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 13:30:28 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0046: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fedf4400 cur 1394044228 last 1394044228 Mar 5 13:30:28 dc2oss07 kernel: : Lustre: 15927:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages Mar 5 13:30:33 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 13:30:33 dc2oss09 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:30:47 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007a_UUID' is not available for connect (no target) Mar 5 13:30:47 dc2oss11 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 13:30:57 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 637920, left: 637920, last alive is 52 secs ago Mar 5 13:30:57 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 637920, left: 637920, last alive is 52 secs ago Mar 5 13:31:26 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0020_UUID' is not available for connect (no target) Mar 5 13:31:26 dc2oss03 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:31:39 dc2oss12 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f8c37800 cur 1394044299 last 1394044299 Mar 5 13:31:39 dc2oss12 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 13:31:40 dc2oss09 kernel: : Lustre: 16453:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394044300 last 1394044300 Mar 5 13:31:40 dc2oss09 kernel: : Lustre: 16453:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 141 previous similar messages Mar 5 13:31:43 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:31:43 dc2oss10 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 13:32:09 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d432800 cur 1394044329 last 1394044329 Mar 5 13:32:09 dc2oss15 kernel: : Lustre: 15726:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 13:32:10 dc2oss13 kernel: : Lustre: 14415:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394044330 last 1394044330 Mar 5 13:32:10 dc2oss13 kernel: : Lustre: 14415:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 13:32:24 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 63424, left: 63424, last alive is 37 secs ago Mar 5 13:32:24 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 63424, left: 63424, last alive is 37 secs ago Mar 5 13:32:27 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818028ec800 cur 1394044347 last 1394044347 Mar 5 13:32:27 dc2oss04 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 13:32:43 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004f_UUID' is not available for connect (no target) Mar 5 13:32:43 dc2oss07 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 13:32:56 gc1 kernel: : Lustre: 8518:0:(mdt_xattr.c:377:mdt_reint_setxattr()) ltest-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200000bd2:0xb2:0x0]] [valid 68719476736] Mar 5 13:33:16 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 906016, left: 906016, last alive is 51 secs ago Mar 5 13:33:16 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 906016, left: 906016, last alive is 51 secs ago Mar 5 13:33:16 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 942336, left: 942336, last alive is 35 secs ago Mar 5 13:33:16 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 942336, left: 942336, last alive is 35 secs ago Mar 5 13:33:29 dc2oss14 kernel: : Lustre: 14587:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0092: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182737d400 cur 1394044409 last 1394044409 Mar 5 13:33:29 dc2oss14 kernel: : Lustre: 14587:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 13:34:10 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t41287867 exp ffff88182be55400 cur 1394044450 last 1394044450 Mar 5 13:34:10 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 13:34:23 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 13:34:23 dc2oss08 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 13:34:43 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881826ba4400 cur 1394044483 last 1394044483 Mar 5 13:34:43 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 83 previous similar messages Mar 5 13:34:53 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 13:34:53 dc2oss14 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:34:59 dc2oss05 kernel: : Lustre: 15851:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0031: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881828419800 cur 1394044499 last 1394044499 Mar 5 13:34:59 dc2oss05 kernel: : Lustre: 15851:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 13:34:53 dc2oss01 kernel: : Lustre: 20209:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/1), not sending early reply Mar 5 13:34:53 dc2oss01 kernel: : Lustre: 20209:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages Mar 5 13:34:54 dc2oss01 kernel: : Lustre: 20308:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/1), not sending early reply Mar 5 13:34:58 dc2oss01 kernel: : LustreError: 15212:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 604+0s req@ffff8806e1ed2800 x1461680520983420/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394044498 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:34:58 dc2oss01 kernel: : LustreError: 15212:0:(ldlm_lib.c:2665:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:34:58 dc2oss01 kernel: : Lustre: dc2-OST0005: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:34:58 dc2oss01 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:34:58 dc2oss01 kernel: : Lustre: Service thread pid 15212 completed after 604.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:34:58 dc2oss01 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:34:59 dc2oss01 kernel: : LustreError: 19586:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 603+0s req@ffff880beb981850 x1461680520983434/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394044499 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:34:59 dc2oss01 kernel: : Lustre: dc2-OST0005: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:34:59 dc2oss01 kernel: : Lustre: Service thread pid 19586 completed after 603.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:35:01 dc2oss04 kernel: : Lustre: dc2-OST0024: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:35:01 dc2oss04 kernel: : Lustre: Skipped 122 previous similar messages Mar 5 13:35:17 dc2oss10 kernel: : Lustre: 16511:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394044517 last 1394044517 Mar 5 13:35:17 dc2oss10 kernel: : Lustre: 16511:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 13:35:18 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0097_UUID' is not available for connect (no target) Mar 5 13:35:18 dc2oss16 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 13:35:29 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:35:29 dc2oss02 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 13:35:29 dc2oss02 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394044529 last 1394044529 Mar 5 13:35:29 dc2oss02 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 13:35:44 dc2oss03 kernel: : Lustre: dc2-OST001c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:35:44 dc2oss03 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 13:35:44 dc2oss03 kernel: : Lustre: 16917:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0df95000 cur 1394044544 last 1394044544 Mar 5 13:35:44 dc2oss03 kernel: : Lustre: 16917:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 13:35:25 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 13:35:25 dc2oss01 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 13:35:49 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 13:35:49 dc2oss02 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 13:35:52 dc2oss06 kernel: : Lustre: 14605:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-55), not sending early reply Mar 5 13:35:57 dc2oss06 kernel: : LustreError: 7076:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 652+0s req@ffff8808fab02000 x1461680520983332/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394044557 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:35:57 dc2oss06 kernel: : Lustre: dc2-OST003d: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:35:57 dc2oss06 kernel: : Lustre: Skipped 4 previous similar messages Mar 5 13:35:57 dc2oss06 kernel: : Lustre: Service thread pid 7076 completed after 659.95s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:35:55 dc2oss14 kernel: : Lustre: dc2-OST0091: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:35:55 dc2oss14 kernel: : Lustre: Skipped 114 previous similar messages Mar 5 13:35:53 dc2oss04 kernel: : Lustre: 16237:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-55), not sending early reply Mar 5 13:35:58 dc2oss04 kernel: : LustreError: 20286:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 660+0s req@ffff880b0a726c00 x1461680520983402/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394044558 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:35:58 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:35:58 dc2oss04 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:35:58 dc2oss04 kernel: : Lustre: Service thread pid 20286 completed after 660.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:35:59 dc2oss04 kernel: : Lustre: 16219:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-55), not sending early reply Mar 5 13:36:09 dc2oss16 kernel: : Lustre: dc2-OST00a5: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:36:09 dc2oss16 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 13:36:04 dc2oss04 kernel: : LustreError: 20469:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk PUT after 660+0s req@ffff880b0d348400 x1461680520983657/t0(0) o3->e9445b99-ed29-0186-e579-ce6798a7bd28@149.165.229.59@tcp:0/0 lens 448/400 e 3 to 0 dl 1394044564 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:36:04 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with e9445b99-ed29-0186-e579-ce6798a7bd28 (at 149.165.229.59@tcp), client will retry: rc -110 Mar 5 13:36:04 dc2oss04 kernel: : Lustre: Service thread pid 20469 completed after 660.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:36:19 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a5_UUID' is not available for connect (no target) Mar 5 13:36:19 dc2oss15 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 13:36:18 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd3306400 cur 1394044578 last 1394044578 Mar 5 13:36:18 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 120 previous similar messages Mar 5 13:36:28 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 13:36:28 dc2oss05 kernel: : LustreError: Skipped 17 previous similar messages Mar 5 13:36:36 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0091_UUID' is not available for connect (no target) Mar 5 13:36:36 dc2oss13 kernel: : LustreError: Skipped 34 previous similar messages Mar 5 13:36:37 dc2oss13 kernel: : Lustre: dc2-OST0084: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:36:37 dc2oss13 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 13:36:54 dc2oss09 kernel: : Lustre: dc2-OST0056: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:36:54 dc2oss09 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 13:36:44 dc2oss01 kernel: : Lustre: dc2-OST0001: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:36:44 dc2oss01 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 13:36:44 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88180e77a400 cur 1394044604 last 1394044604 Mar 5 13:36:44 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 13:37:03 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST0032_UUID' is not available for connect (no target) Mar 5 13:37:03 dc2oss06 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 13:36:57 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:36:57 dc2mds01 kernel: : Lustre: Skipped 27 previous similar messages Mar 5 13:36:57 dc2mds01 kernel: : Lustre: 11185:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394044617 last 1394044617 Mar 5 13:36:57 dc2mds01 kernel: : Lustre: 11185:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 27 previous similar messages Mar 5 13:37:17 dc2oss15 kernel: : Lustre: dc2-OST009a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:37:17 dc2oss15 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 13:37:27 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:37:27 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 19 previous similar messages Mar 5 13:37:55 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881827011c00 cur 1394044675 last 1394044675 Mar 5 13:37:55 dc2oss08 kernel: : Lustre: 15891:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:38:24 dc2oss11 kernel: : Lustre: dc2-OST0069: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:38:24 dc2oss11 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:38:49 dc2oss08 kernel: : Lustre: dc2-OST004d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:38:49 dc2oss08 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 13:38:48 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0073_UUID' is not available for connect (no target) Mar 5 13:38:48 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:38:54 dc2oss06 kernel: : Lustre: dc2-OST003e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:38:54 dc2oss06 kernel: : Lustre: Skipped 75 previous similar messages Mar 5 13:38:53 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001a_UUID' is not available for connect (no target) Mar 5 13:38:53 dc2oss04 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:39:10 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:39:10 dc2oss12 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 13:39:19 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:39:19 dc2oss05 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 13:39:22 dc2oss11 kernel: : Lustre: 16209:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394044755/real 0] req@ffff8815ee808800 x1456766131033343/t0(0) o104->dc2-OST0073@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394044762 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:39:22 dc2oss11 kernel: : LustreError: 138-a: dc2-OST0073: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 13:39:22 dc2oss11 kernel: : LustreError: 14476:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880bd4c23000 x1456766131033373/t0(0) o105->dc2-OST0073@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 Mar 5 13:39:22 dc2oss11 kernel: : LustreError: 14476:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.206@tcp) returned 0 from completion AST ns: filter-dc2-OST0073_UUID lock: ffff88175f8c96c0/0x309c239b3af12a1d lrc: 3/0,0 mode: PW/PW res: 21885599/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->81919) flags: 0x0 remote: 0xfd05479c59454aef expref: 567 pid: 16209 timeout 0 Mar 5 13:39:48 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0057_UUID' is not available for connect (no target) Mar 5 13:39:48 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:40:12 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f953:0x34e:0x0]] [valid 68719476736] Mar 5 13:40:12 dc2mds01 kernel: : Lustre: 11191:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 65 previous similar messages Mar 5 13:40:28 dc2oss07 kernel: : Lustre: dc2-OST0047: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:40:28 dc2oss07 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:40:28 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1394044828 last 1394044828 Mar 5 13:40:28 dc2oss07 kernel: : Lustre: 15872:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:40:46 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0061_UUID' is not available for connect (no target) Mar 5 13:40:46 dc2oss09 kernel: : LustreError: Skipped 14 previous similar messages Mar 5 13:41:00 dc2mds01 kernel: : LustreError: 6341:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 232072699489, new: 232072699488 replay: 0. see LU-617. Mar 5 13:41:31 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0028_UUID' is not available for connect (no target) Mar 5 13:41:31 dc2oss03 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:41:15 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) Mar 5 13:41:15 dc2oss11 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:41:41 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0077: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817cce71400 cur 1394044901 last 1394044901 Mar 5 13:41:41 dc2oss12 kernel: : Lustre: 14490:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 100 previous similar messages Mar 5 13:41:45 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0059: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c087a3800 cur 1394044905 last 1394044905 Mar 5 13:41:45 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 13:41:47 dc2oss10 kernel: : Lustre: dc2-OST0060: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:41:47 dc2oss10 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 13:42:12 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bee795000 cur 1394044932 last 1394044932 Mar 5 13:42:12 dc2oss15 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 13:42:21 dc2oss13 kernel: : Lustre: 14415:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394044941 last 1394044941 Mar 5 13:42:21 dc2oss13 kernel: : Lustre: 14415:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 13:42:21 dc2oss11 kernel: : Lustre: 25352:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 13:42:26 dc2oss11 kernel: : LustreError: 8788:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bc2076000 x1442912237026126/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394044946 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:42:26 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:42:26 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:42:26 dc2oss11 kernel: : Lustre: 25452:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 13:42:29 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 7 active RPCs Mar 5 13:42:29 dc2oss11 kernel: : LustreError: 25469:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bd5fa4c00 x1442912237026358/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394044961 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:42:29 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:42:29 dc2oss11 kernel: : LustreError: 25399:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880c0effe000 x1442912237027284/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045056 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:42:29 dc2oss11 kernel: : LustreError: 8664:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880feb4e0850 x1442912237027521/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045100 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:42:29 dc2oss11 kernel: : LustreError: 8664:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:42:29 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394044949 last 1394044949 Mar 5 13:42:29 dc2oss04 kernel: : Lustre: 16742:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:43:01 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0052_UUID' is not available for connect (no target) Mar 5 13:43:01 dc2oss07 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:43:30 dc2oss14 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bebf43400 cur 1394045010 last 1394045010 Mar 5 13:43:30 dc2oss14 kernel: : Lustre: 14478:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 13:44:12 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 431840, left: 431840, last alive is 37 secs ago Mar 5 13:44:12 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 431840, left: 431840, last alive is 37 secs ago Mar 5 13:44:14 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a5: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05f69c00 cur 1394045054 last 1394045054 Mar 5 13:44:14 dc2oss16 kernel: : Lustre: 15745:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 13:44:22 dc2oss08 kernel: : Lustre: 18687:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394045055/real 1394045055] req@ffff880bd5e9f000 x1456766190366350/t0(0) o105->dc2-OST0052@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394045062 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:44:22 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0052: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:44:38 dc2oss11 kernel: : Lustre: 14537:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394045071/real 1394045071] req@ffff880f8eb9b400 x1456766131034687/t0(0) o104->dc2-OST006d@149.165.226.206@tcp:15/16 lens 296/192 e 0 to 1 dl 1394045078 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:44:38 dc2oss11 kernel: : LustreError: 138-a: dc2-OST006d: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 13:44:38 dc2oss11 kernel: : LustreError: 14476:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880bd5666800 x1456766131034706/t0(0) o105->dc2-OST006d@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 Mar 5 13:44:38 dc2oss11 kernel: : LustreError: 14476:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.206@tcp) returned 0 from completion AST ns: filter-dc2-OST006d_UUID lock: ffff8817e3c34d80/0x309c239b3af214ab lrc: 3/0,0 mode: PW/PW res: 22377135/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->12287) flags: 0x0 remote: 0xfd05479c5958c485 expref: 547 pid: 14537 timeout 0 Mar 5 13:44:39 dc2oss08 kernel: : Lustre: 18687:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394045071/real 1394045071] req@ffff88006dfe8c00 x1456766190366422/t0(0) o105->dc2-OST0053@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394045079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:44:39 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0053: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:44:39 dc2oss08 kernel: : LustreError: 27727:0:(ldlm_lib.c:2669:target_bulk_io()) @@@ Eviction on bulk PUT req@ffff880bd3213800 x1461691364730019/t0(0) o3->00795451-8f38-1361-7a6d-ab2d6c88925e@149.165.226.206@tcp:0/0 lens 448/400 e 0 to 0 dl 1394045097 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:44:39 dc2oss08 kernel: : LustreError: 27727:0:(ldlm_lib.c:2669:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:44:39 dc2oss08 kernel: : Lustre: dc2-OST0053: Bulk IO read error with 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp), client will retry: rc -107 Mar 5 13:44:40 dc2oss08 kernel: : Lustre: 7147:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394045071/real 1394045071] req@ffff8806e6c85000 x1456766190366423/t0(0) o105->dc2-OST0050@149.165.226.206@tcp:15/16 lens 344/192 e 0 to 1 dl 1394045080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 13:44:40 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0050: A client on nid 149.165.226.206@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 13:44:47 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c112d0800 cur 1394045087 last 1394045087 Mar 5 13:44:47 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 13:44:50 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 198880, left: 198880, last alive is 37 secs ago Mar 5 13:44:50 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 198880, left: 198880, last alive is 37 secs ago Mar 5 13:45:02 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0087_UUID' is not available for connect (no target) Mar 5 13:45:02 dc2oss14 kernel: : LustreError: Skipped 30 previous similar messages Mar 5 13:45:04 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394045104 last 1394045104 Mar 5 13:45:04 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 117 previous similar messages Mar 5 13:45:21 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bfd6ed800 cur 1394045121 last 1394045121 Mar 5 13:45:21 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 94 previous similar messages Mar 5 13:45:32 dc2oss02 kernel: : Lustre: dc2-OST000f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:45:32 dc2oss02 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 13:45:32 dc2oss02 kernel: : Lustre: 15930:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c01a83800 cur 1394045132 last 1394045132 Mar 5 13:45:32 dc2oss02 kernel: : Lustre: 15930:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:45:19 dc2oss04 kernel: : Lustre: dc2-OST0026: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:45:19 dc2oss04 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 13:45:35 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0049_UUID' is not available for connect (no target) Mar 5 13:45:35 dc2oss08 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 13:46:14 dc2oss03 kernel: : Lustre: dc2-OST001e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:46:14 dc2oss03 kernel: : Lustre: Skipped 92 previous similar messages Mar 5 13:46:14 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf8144400 cur 1394045174 last 1394045174 Mar 5 13:46:14 dc2oss03 kernel: : Lustre: 16604:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 13:45:59 dc2oss14 kernel: : Lustre: dc2-OST008f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:45:59 dc2oss14 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 13:45:59 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009d_UUID' is not available for connect (no target) Mar 5 13:45:59 dc2oss16 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 13:46:20 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST0003_UUID' is not available for connect (no target) Mar 5 13:46:20 dc2oss02 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 13:46:13 dc2oss16 kernel: : Lustre: dc2-OST00a0: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:46:13 dc2oss16 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 13:46:19 dc2oss11 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881825b13400 cur 1394045179 last 1394045179 Mar 5 13:46:19 dc2oss11 kernel: : Lustre: 14580:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 103 previous similar messages Mar 5 13:46:34 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST009e_UUID' is not available for connect (no target) Mar 5 13:46:34 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:46:28 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 13:46:28 dc2oss01 kernel: : LustreError: Skipped 33 previous similar messages Mar 5 13:46:37 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003c_UUID' is not available for connect (no target) Mar 5 13:46:37 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 13:46:35 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 13:46:35 dc2oss04 kernel: : LustreError: 16248:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880aa5ad1c00 x1461679899211518/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394045229 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:46:35 dc2oss04 kernel: : LustreError: 16248:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:46:35 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:46:35 dc2oss04 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:46:36 dc2oss04 kernel: : LustreError: 20201:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b08588c00 x1461679899211534/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394045232 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:46:36 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 13:46:41 dc2oss13 kernel: : Lustre: dc2-OST0087: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:46:41 dc2oss13 kernel: : Lustre: Skipped 102 previous similar messages Mar 5 13:46:54 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:46:54 dc2oss09 kernel: : Lustre: Skipped 123 previous similar messages Mar 5 13:46:49 dc2oss01 kernel: : Lustre: dc2-OST0004: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:46:49 dc2oss01 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 13:46:49 dc2oss01 kernel: : Lustre: 15435:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880be6cbf400 cur 1394045209 last 1394045209 Mar 5 13:46:49 dc2oss01 kernel: : Lustre: 15435:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 13:47:16 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 13:47:16 dc2oss06 kernel: : LustreError: Skipped 29 previous similar messages Mar 5 13:47:25 dc2oss15 kernel: : Lustre: dc2-OST009d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:47:25 dc2oss15 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 13:47:22 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:47:22 dc2mds01 kernel: : Lustre: Skipped 28 previous similar messages Mar 5 13:47:22 dc2mds01 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394045242 last 1394045242 Mar 5 13:47:22 dc2mds01 kernel: : Lustre: 14548:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 28 previous similar messages Mar 5 13:47:42 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) Mar 5 13:47:42 dc2oss13 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 13:47:47 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 466272, left: 466272, last alive is 43 secs ago Mar 5 13:47:47 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 466272, left: 466272, last alive is 43 secs ago Mar 5 13:48:02 dc2oss04 kernel: : Lustre: dc2-OST0024: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811b3b09c00, cur 1394045282 expire 1394045132 last 1394045055 Mar 5 13:48:02 dc2oss04 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:02 dc2oss04 kernel: : Lustre: dc2-OST0029: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811bff10000, cur 1394045282 expire 1394045132 last 1394045055 Mar 5 13:48:03 dc2oss04 kernel: : Lustre: dc2-OST0026: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811f4ec5400, cur 1394045283 expire 1394045133 last 1394045056 Mar 5 13:48:06 dc2oss04 kernel: : Lustre: dc2-OST0020: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812172e4800, cur 1394045286 expire 1394045136 last 1394045059 Mar 5 13:48:07 dc2oss14 kernel: : Lustre: dc2-OST008c: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f036c3c00, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss14 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:09 dc2oss14 kernel: : Lustre: dc2-OST0089: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bc6fd4400, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:11 dc2oss14 kernel: : Lustre: dc2-OST008a: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811e8c1cc00, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:07 dc2oss09 kernel: : Lustre: dc2-OST005e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092be2c400, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss09 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:09 dc2oss09 kernel: : Lustre: dc2-OST005c: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88139c3fb000, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:11 dc2oss09 kernel: : Lustre: dc2-OST005d: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814fd5f5000, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:12 dc2oss09 kernel: : Lustre: dc2-OST0057: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880600af7400, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:03 dc2oss08 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c257a8400 cur 1394045283 last 1394045283 Mar 5 13:48:03 dc2oss08 kernel: : Lustre: 16079:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 77 previous similar messages Mar 5 13:48:11 dc2oss08 kernel: : Lustre: dc2-OST004c: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c9f33bc00, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:11 dc2oss08 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:12 dc2oss08 kernel: : Lustre: dc2-OST004a: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f9dab5000, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:13 dc2oss08 kernel: : Lustre: dc2-OST004f: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd37fac00, cur 1394045293 expire 1394045143 last 1394045066 Mar 5 13:47:55 dc2mds01 kernel: : Lustre: MGS: haven't heard from client cc6c84d8-c55c-b2d0-c35c-f36201c46340 (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882fe69e4c00, cur 1394045275 expire 1394045125 last 1394045048 Mar 5 13:48:07 dc2oss10 kernel: : Lustre: dc2-OST0061: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e1ac85c00, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss10 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:09 dc2oss10 kernel: : Lustre: dc2-OST0068: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd7b1f000, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:11 dc2oss10 kernel: : Lustre: dc2-OST0062: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c9f1fd000, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:15 dc2oss10 kernel: : Lustre: dc2-OST005f: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d22e4a000, cur 1394045295 expire 1394045145 last 1394045068 Mar 5 13:48:07 dc2oss05 kernel: : Lustre: dc2-OST002f: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d5a50800, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss05 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:11 dc2oss05 kernel: : Lustre: dc2-OST002e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88179a0b3000, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:12 dc2oss05 kernel: : Lustre: dc2-OST0030: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e199c9400, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:14 dc2oss05 kernel: : Lustre: dc2-OST0031: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88179a0b3c00, cur 1394045294 expire 1394045144 last 1394045067 Mar 5 13:48:07 dc2oss13 kernel: : Lustre: dc2-OST0082: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8d6ca400, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss13 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:07 dc2oss13 kernel: : Lustre: dc2-OST0084: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f62c6f000, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:12 dc2oss13 kernel: : Lustre: dc2-OST0086: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811ba2bc800, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:17 dc2oss13 kernel: : Lustre: dc2-OST007e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e3fc02000, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:07 dc2oss11 kernel: : Lustre: dc2-OST0069: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5dbc2c00, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss11 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:11 dc2oss11 kernel: : Lustre: dc2-OST0073: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd423c400, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:13 dc2oss11 kernel: : Lustre: dc2-OST006f: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd390b800, cur 1394045293 expire 1394045143 last 1394045066 Mar 5 13:48:15 dc2oss11 kernel: : Lustre: dc2-OST006b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf7c6ac00, cur 1394045295 expire 1394045145 last 1394045068 Mar 5 13:48:15 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:07 dc2oss15 kernel: : Lustre: dc2-OST009d: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bce8a6800, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss15 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:07 dc2oss15 kernel: : Lustre: dc2-OST0094: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bbbeeac00, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:11 dc2oss15 kernel: : Lustre: dc2-OST0093: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814c01c4c00, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:12 dc2oss15 kernel: : Lustre: dc2-OST0096: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd4d1a400, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:07 dc2oss06 kernel: : Lustre: dc2-OST0035: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813f6bff800, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss06 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:07 dc2oss06 kernel: : Lustre: dc2-OST003a: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811a35c2000, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:13 dc2oss06 kernel: : Lustre: dc2-OST003b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fed164000, cur 1394045293 expire 1394045143 last 1394045066 Mar 5 13:48:15 dc2oss06 kernel: : Lustre: dc2-OST0038: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813f6bfe800, cur 1394045295 expire 1394045145 last 1394045068 Mar 5 13:48:18 dc2oss02 kernel: : Lustre: dc2-OST000b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d692aa000, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:18 dc2oss02 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:18 dc2oss02 kernel: : Lustre: dc2-OST0013: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880790d2c800, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:19 dc2oss02 kernel: : Lustre: dc2-OST000e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88078e7e7800, cur 1394045299 expire 1394045149 last 1394045072 Mar 5 13:48:21 dc2oss02 kernel: : Lustre: dc2-OST0010: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d7910f000, cur 1394045301 expire 1394045151 last 1394045074 Mar 5 13:48:21 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:24 dc2oss02 kernel: : Lustre: dc2-OST0011: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880790497000, cur 1394045304 expire 1394045154 last 1394045077 Mar 5 13:48:27 dc2oss03 kernel: : Lustre: dc2-OST001a: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806b7630000, cur 1394045307 expire 1394045157 last 1394045080 Mar 5 13:48:27 dc2oss03 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:29 dc2oss03 kernel: : Lustre: dc2-OST001b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814064e6c00, cur 1394045309 expire 1394045159 last 1394045082 Mar 5 13:48:35 dc2oss03 kernel: : Lustre: dc2-OST001c: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005a6cb400, cur 1394045315 expire 1394045165 last 1394045088 Mar 5 13:48:36 dc2oss03 kernel: : Lustre: dc2-OST0015: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814039c8800, cur 1394045316 expire 1394045166 last 1394045089 Mar 5 13:48:36 dc2oss03 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:07 dc2oss12 kernel: : Lustre: dc2-OST0076: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815c8f21800, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss12 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:09 dc2oss12 kernel: : Lustre: dc2-OST0078: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd295b800, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:12 dc2oss12 kernel: : Lustre: dc2-OST007b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811cd880000, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:16 dc2oss12 kernel: : Lustre: dc2-OST0075: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d7bad4800, cur 1394045296 expire 1394045146 last 1394045069 Mar 5 13:48:16 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:00 dc2oss01 kernel: : Lustre: dc2-OST0009: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806e2e71400, cur 1394045280 expire 1394045130 last 1394045053 Mar 5 13:48:00 dc2oss01 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:02 dc2oss01 kernel: : Lustre: dc2-OST0002: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee3114400, cur 1394045282 expire 1394045132 last 1394045055 Mar 5 13:48:05 dc2oss01 kernel: : Lustre: dc2-OST0000: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb4e48000, cur 1394045285 expire 1394045135 last 1394045058 Mar 5 13:48:07 dc2oss01 kernel: : Lustre: dc2-OST0004: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e416df000, cur 1394045287 expire 1394045137 last 1394045060 Mar 5 13:48:07 dc2oss01 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:48:09 dc2oss07 kernel: : Lustre: dc2-OST0041: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815c607ec00, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:09 dc2oss07 kernel: : Lustre: Skipped 10 previous similar messages Mar 5 13:48:09 dc2oss07 kernel: : Lustre: dc2-OST0046: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88114f609c00, cur 1394045289 expire 1394045139 last 1394045062 Mar 5 13:48:10 dc2oss07 kernel: : Lustre: dc2-OST0045: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812398f6c00, cur 1394045290 expire 1394045140 last 1394045063 Mar 5 13:48:15 dc2oss07 kernel: : Lustre: dc2-OST0048: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2b6f8c00, cur 1394045295 expire 1394045145 last 1394045068 Mar 5 13:48:15 dc2oss07 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:48:17 dc2oss07 kernel: : Lustre: dc2-OST0040: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d118d1400, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:17 dc2oss07 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:48:11 dc2oss16 kernel: : Lustre: dc2-OST00a2: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ceeb81400, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:11 dc2oss16 kernel: : Lustre: Skipped 9 previous similar messages Mar 5 13:48:11 dc2oss16 kernel: : Lustre: dc2-OST00a5: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814a5fe9c00, cur 1394045291 expire 1394045141 last 1394045064 Mar 5 13:48:12 dc2oss16 kernel: : Lustre: dc2-OST00a4: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811897e0000, cur 1394045292 expire 1394045142 last 1394045065 Mar 5 13:48:15 dc2oss16 kernel: : Lustre: dc2-OST009e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e7b661400, cur 1394045295 expire 1394045145 last 1394045068 Mar 5 13:48:15 dc2oss16 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:18 dc2oss16 kernel: : Lustre: dc2-OST00a6: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813670ef800, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:13 dc2oss14 kernel: : Lustre: dc2-OST008d: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88118f4a4800, cur 1394045293 expire 1394045143 last 1394045066 Mar 5 13:48:13 dc2oss14 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:17 dc2oss14 kernel: : Lustre: dc2-OST008f: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811e8c1c000, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:17 dc2oss14 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:14 dc2oss08 kernel: : Lustre: dc2-OST004b: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd4940800, cur 1394045294 expire 1394045144 last 1394045067 Mar 5 13:48:17 dc2oss08 kernel: : Lustre: dc2-OST0051: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881083b9dc00, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:17 dc2oss08 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:10 dc2oss04 kernel: : Lustre: dc2-OST0027: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88124a13e800, cur 1394045290 expire 1394045140 last 1394045063 Mar 5 13:48:16 dc2oss09 kernel: : Lustre: dc2-OST0059: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816ae301800, cur 1394045296 expire 1394045146 last 1394045069 Mar 5 13:48:16 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:17 dc2oss06 kernel: : Lustre: dc2-OST0037: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5689d800, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:17 dc2oss06 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:48:17 dc2oss15 kernel: : Lustre: dc2-OST0099: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bce833400, cur 1394045297 expire 1394045147 last 1394045070 Mar 5 13:48:17 dc2oss15 kernel: : Lustre: Skipped 2 previous similar messages Mar 5 13:48:06 dc2mds01 kernel: : Lustre: dc2-MDT0000: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882f8887f400, cur 1394045286 expire 1394045136 last 1394045059 Mar 5 13:48:18 dc2oss10 kernel: : Lustre: dc2-OST0064: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e87284400, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:18 dc2oss10 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:48:18 dc2oss05 kernel: : Lustre: dc2-OST002a: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fa330f400, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:18 dc2oss05 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:48:18 dc2oss11 kernel: : Lustre: dc2-OST006e: haven't heard from client 00795451-8f38-1361-7a6d-ab2d6c88925e (at 149.165.226.206@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880edab5a800, cur 1394045298 expire 1394045148 last 1394045071 Mar 5 13:48:18 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:48:18 dc2oss13 kernel: : LustreError: 138-a: dc2-OST0080: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -4 Mar 5 13:48:18 dc2oss13 kernel: : LustreError: 138-a: dc2-OST0083: A client on nid 149.165.226.206@tcp was evicted due to a lock blocking callback time out: rc -4 Mar 5 13:48:44 dc2oss11 kernel: : Lustre: dc2-OST006f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:48:44 dc2oss11 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 13:48:55 dc2oss06 kernel: : Lustre: dc2-OST0039: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:48:55 dc2oss06 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 13:48:59 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:48:59 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 15 previous similar messages Mar 5 13:49:14 dc2oss12 kernel: : Lustre: dc2-OST007c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:49:14 dc2oss12 kernel: : Lustre: Skipped 97 previous similar messages Mar 5 13:49:19 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006b_UUID' is not available for connect (no target) Mar 5 13:49:19 dc2oss12 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:49:18 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:49:18 dc2oss08 kernel: : Lustre: Skipped 76 previous similar messages Mar 5 13:49:24 dc2oss05 kernel: : Lustre: dc2-OST0032: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:49:24 dc2oss05 kernel: : Lustre: Skipped 121 previous similar messages Mar 5 13:50:03 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 13:50:03 dc2oss10 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 13:50:19 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST001c_UUID' is not available for connect (no target) Mar 5 13:50:19 dc2oss04 kernel: : LustreError: Skipped 18 previous similar messages Mar 5 13:50:13 dc2mds01 kernel: : Lustre: 14548:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0fafe:0x35:0x0]] [valid 68719476736] Mar 5 13:50:13 dc2mds01 kernel: : Lustre: 14548:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 49 previous similar messages Mar 5 13:50:30 dc2oss07 kernel: : Lustre: dc2-OST0047: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:50:30 dc2oss07 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 13:50:30 dc2oss07 kernel: : Lustre: 14342:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817fb92dc00 cur 1394045430 last 1394045430 Mar 5 13:50:30 dc2oss07 kernel: : Lustre: 14342:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 13:51:07 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0068_UUID' is not available for connect (no target) Mar 5 13:51:07 dc2oss09 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:52:02 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0023_UUID' is not available for connect (no target) Mar 5 13:52:02 dc2oss03 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 13:51:46 dc2oss09 kernel: : Lustre: 16311:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182d52ec00 cur 1394045506 last 1394045506 Mar 5 13:51:46 dc2oss09 kernel: : Lustre: 16311:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 13:51:48 dc2oss10 kernel: : Lustre: dc2-OST0068: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:51:48 dc2oss10 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 13:51:48 dc2oss12 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394045508 last 1394045508 Mar 5 13:51:48 dc2oss12 kernel: : Lustre: 14560:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 13:52:24 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 13:52:24 dc2oss11 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:52:32 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd47bac00 cur 1394045552 last 1394045552 Mar 5 13:52:32 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 97 previous similar messages Mar 5 13:52:29 dc2oss04 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t40930343 exp ffff8817f3d56800 cur 1394045549 last 1394045549 Mar 5 13:52:29 dc2oss04 kernel: : Lustre: 16100:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 146 previous similar messages Mar 5 13:52:34 dc2oss15 kernel: : Lustre: 16356:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394045554 last 1394045554 Mar 5 13:52:34 dc2oss15 kernel: : Lustre: 16356:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 122 previous similar messages Mar 5 13:53:02 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 9312, left: 9312, last alive is 5 secs ago Mar 5 13:53:02 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 9312, left: 9312, last alive is 5 secs ago Mar 5 13:53:05 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004b_UUID' is not available for connect (no target) Mar 5 13:53:05 dc2oss07 kernel: : LustreError: Skipped 22 previous similar messages Mar 5 13:53:32 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0091: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c04587000 cur 1394045612 last 1394045612 Mar 5 13:53:32 dc2oss14 kernel: : Lustre: 15622:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 99 previous similar messages Mar 5 13:54:22 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bd1db0000 cur 1394045662 last 1394045662 Mar 5 13:54:22 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 108 previous similar messages Mar 5 13:54:49 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181ae1d400 cur 1394045689 last 1394045689 Mar 5 13:54:49 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 13:55:06 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST007f_UUID' is not available for connect (no target) Mar 5 13:55:06 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 13:55:24 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0f762000 cur 1394045724 last 1394045724 Mar 5 13:55:24 dc2oss10 kernel: : Lustre: 16363:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 92 previous similar messages Mar 5 13:55:25 dc2oss11 kernel: : Lustre: Service thread pid 8673 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 13:55:25 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 13:55:25 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394045725.8673 Mar 5 13:55:21 dc2oss04 kernel: : Lustre: dc2-OST0027: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:55:21 dc2oss04 kernel: : Lustre: Skipped 150 previous similar messages Mar 5 13:55:37 dc2oss02 kernel: : Lustre: dc2-OST0012: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:55:37 dc2oss02 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 13:55:37 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0012: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf3a7a800 cur 1394045737 last 1394045737 Mar 5 13:55:37 dc2oss02 kernel: : Lustre: 16328:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 81 previous similar messages Mar 5 13:55:26 dc2oss05 kernel: : Lustre: 16023:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0032: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bff047800 cur 1394045726 last 1394045726 Mar 5 13:55:26 dc2oss05 kernel: : Lustre: 16023:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages Mar 5 13:55:47 dc2oss11 kernel: : Lustre: 8750:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 13:55:51 dc2oss11 kernel: : Lustre: 14610:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Mar 5 13:55:51 dc2oss11 kernel: : Lustre: 14610:0:(service.c:1035:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Mar 5 13:55:52 dc2oss11 kernel: : LustreError: 25304:0:(ldlm_lib.c:2665:target_bulk_io()) @@@ timeout on bulk GET after 755+0s req@ffff880bf383e000 x1442912237034335/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045752 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:55:52 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:55:52 dc2oss11 kernel: : Lustre: Skipped 6 previous similar messages Mar 5 13:55:54 dc2oss11 kernel: : Lustre: dc2-OST006e: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) refused reconnection, still busy with 6 active RPCs Mar 5 13:55:54 dc2oss11 kernel: : LustreError: 8743:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bf3870400 x1442912237034786/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045792 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:55:54 dc2oss11 kernel: : LustreError: 8743:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:55:54 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:55:54 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message Mar 5 13:55:54 dc2oss11 kernel: : Lustre: Service thread pid 8673 completed after 229.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 5 13:55:54 dc2oss11 kernel: : LustreError: 8805:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff8817faeda050 x1442912237034556/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045756 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:55:54 dc2oss11 kernel: : LustreError: 8805:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 13:55:55 dc2oss11 kernel: : Lustre: dc2-OST006e: Bulk IO write error with fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp), client will retry: rc -110 Mar 5 13:55:55 dc2oss11 kernel: : Lustre: Skipped 3 previous similar messages Mar 5 13:55:55 dc2oss11 kernel: : LustreError: 8732:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff880bf3b76400 x1442912237034787/t0(0) o4->fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp:0/0 lens 448/416 e 0 to 0 dl 1394045794 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 13:55:55 dc2oss11 kernel: : LustreError: 8732:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages Mar 5 13:56:16 dc2oss03 kernel: : Lustre: dc2-OST001b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:16 dc2oss03 kernel: : Lustre: Skipped 118 previous similar messages Mar 5 13:56:16 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c026ac800 cur 1394045776 last 1394045776 Mar 5 13:56:16 dc2oss03 kernel: : Lustre: 16913:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 119 previous similar messages Mar 5 13:55:55 dc2xfer04 kernel: : Lustre: 2714:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 13:55:55 dc2xfer04 kernel: : Lustre: 2714:0:(o2iblnd_cb.c:380:kiblnd_handle_rx()) PUT_NACK from 10.10.0.11@o2ib Mar 5 13:56:01 dc2oss14 kernel: : Lustre: dc2-OST0089: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:01 dc2oss14 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 13:56:09 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST0048_UUID' is not available for connect (no target) Mar 5 13:56:09 dc2oss08 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:56:11 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST0093_UUID' is not available for connect (no target) Mar 5 13:56:11 dc2oss16 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 13:56:14 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:14 dc2oss16 kernel: : Lustre: Skipped 107 previous similar messages Mar 5 13:56:32 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 13:56:32 dc2oss02 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:56:25 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c0e449000 cur 1394045785 last 1394045785 Mar 5 13:56:25 dc2oss11 kernel: : Lustre: 14549:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 13:56:47 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0035_UUID' is not available for connect (no target) Mar 5 13:56:47 dc2oss05 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 13:56:49 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a1_UUID' is not available for connect (no target) Mar 5 13:56:49 dc2oss15 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 13:56:51 dc2oss13 kernel: : Lustre: dc2-OST0082: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:51 dc2oss13 kernel: : Lustre: Skipped 101 previous similar messages Mar 5 13:56:54 dc2oss09 kernel: : Lustre: dc2-OST005a: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:54 dc2oss09 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 13:56:49 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0010_UUID' is not available for connect (no target) Mar 5 13:56:49 dc2oss01 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 13:56:52 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:56:52 dc2oss01 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 13:56:52 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394045812 last 1394045812 Mar 5 13:56:52 dc2oss01 kernel: : Lustre: 15111:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 96 previous similar messages Mar 5 13:57:30 dc2oss15 kernel: : Lustre: dc2-OST0095: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:57:30 dc2oss15 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 13:57:29 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:57:29 dc2mds01 kernel: : Lustre: Skipped 23 previous similar messages Mar 5 13:57:29 dc2mds01 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t216355239383 exp ffff881825bc6800 cur 1394045849 last 1394045849 Mar 5 13:57:29 dc2mds01 kernel: : Lustre: 14557:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 23 previous similar messages Mar 5 13:57:42 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008a_UUID' is not available for connect (no target) Mar 5 13:57:42 dc2oss13 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 13:58:05 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t35727150 exp ffff880c049ca800 cur 1394045885 last 1394045885 Mar 5 13:58:05 dc2oss08 kernel: : Lustre: 15968:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 13:58:59 dc2oss06 kernel: : Lustre: dc2-OST0037: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:58:59 dc2oss06 kernel: : Lustre: Skipped 110 previous similar messages Mar 5 13:59:02 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 13:59:02 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 24 previous similar messages Mar 5 13:59:14 dc2oss12 kernel: : Lustre: dc2-OST0076: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:59:14 dc2oss12 kernel: : Lustre: Skipped 96 previous similar messages Mar 5 13:59:21 dc2oss11 kernel: : Lustre: dc2-OST006b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:59:21 dc2oss11 kernel: : Lustre: Skipped 108 previous similar messages Mar 5 13:59:35 dc2oss05 kernel: : Lustre: dc2-OST002b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 13:59:35 dc2oss05 kernel: : Lustre: Skipped 104 previous similar messages Mar 5 14:00:07 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002b_UUID' is not available for connect (no target) Mar 5 14:00:07 dc2oss06 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 14:00:13 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST0072_UUID' is not available for connect (no target) Mar 5 14:00:13 dc2oss12 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 14:00:26 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST0059_UUID' is not available for connect (no target) Mar 5 14:00:26 dc2oss10 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 14:00:31 dc2oss07 kernel: : Lustre: dc2-OST0045: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:00:31 dc2oss07 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 14:00:31 dc2oss07 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff881806538000 cur 1394046031 last 1394046031 Mar 5 14:00:31 dc2oss07 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 89 previous similar messages Mar 5 14:00:54 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:00:54 dc2oss08 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 14:01:09 dc2oss02 kernel: : Lustre: dc2-OST0011: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 14:01:10 dc2oss02 kernel: : LustreError: 20714:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88078fbe5800 x1461679899231353/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046097 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:10 dc2oss02 kernel: : LustreError: 20714:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 6 previous similar messages Mar 5 14:01:10 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:10 dc2oss02 kernel: : Lustre: Skipped 6 previous similar messages Mar 5 14:01:10 dc2oss02 kernel: : LustreError: 20654:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf79e7850 x1461679899231392/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046100 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:10 dc2oss02 kernel: : LustreError: 20654:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 14:01:10 dc2oss02 kernel: : Lustre: dc2-OST0011: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:10 dc2oss02 kernel: : Lustre: Skipped 1 previous similar message Mar 5 14:01:10 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 3 active RPCs Mar 5 14:01:10 dc2oss10 kernel: : LustreError: 8762:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880915bbf000 x1461679899231396/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046109 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:10 dc2oss10 kernel: : LustreError: 8762:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 14:01:10 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:10 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message Mar 5 14:01:10 dc2oss10 kernel: : LustreError: 15975:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be136c800 x1461679899231400/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046109 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:10 dc2oss10 kernel: : Lustre: dc2-OST0065: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:01 dc2oss14 kernel: : Lustre: dc2-OST0090: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 2 active RPCs Mar 5 14:01:01 dc2oss14 kernel: : LustreError: 32031:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc877f800 x1461679899231371/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046097 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:01 dc2oss14 kernel: : Lustre: dc2-OST0090: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:01 dc2oss14 kernel: : LustreError: 22641:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be1005c00 x1461679899231366/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046096 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:01 dc2oss14 kernel: : Lustre: dc2-OST0090: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:09 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0060_UUID' is not available for connect (no target) Mar 5 14:01:09 dc2oss09 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 14:01:00 dc2oss04 kernel: : Lustre: dc2-OST0028: Client 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp) refused reconnection, still busy with 5 active RPCs Mar 5 14:01:01 dc2oss04 kernel: : LustreError: 16245:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880c04d05000 x1461679899231346/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046100 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:01 dc2oss04 kernel: : LustreError: 16245:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message Mar 5 14:01:01 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:01:01 dc2oss04 kernel: : Lustre: Skipped 1 previous similar message Mar 5 14:01:01 dc2oss04 kernel: : LustreError: 20196:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880b0ec16000 x1461679899231341/t0(0) o3->48fdc445-9bd9-c49b-264b-0ded2da1a98f@149.165.229.64@tcp:0/0 lens 448/400 e 1 to 0 dl 1394046099 ref 1 fl Interpret:/0/0 rc 0/0 Mar 5 14:01:01 dc2oss04 kernel: : Lustre: dc2-OST0028: Bulk IO read error with 48fdc445-9bd9-c49b-264b-0ded2da1a98f (at 149.165.229.64@tcp), client will retry: rc -110 Mar 5 14:00:55 dc2mds01 kernel: : Lustre: 10011:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f792:0xb9:0x0]] [valid 68719476736] Mar 5 14:00:55 dc2mds01 kernel: : Lustre: 10011:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 34 previous similar messages Mar 5 14:01:33 dc2oss03 kernel: : Lustre: 17094:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394046086/real 0] req@ffff8812269be800 x1461670870689310/t0(0) o104->dc2-OST001a@149.165.229.26@tcp:15/16 lens 296/192 e 0 to 1 dl 1394046093 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 14:01:33 dc2oss03 kernel: : LustreError: 138-a: dc2-OST001a: A client on nid 149.165.229.26@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 14:01:34 dc2oss03 kernel: : LustreError: 24084:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 149.165.229.26@tcp arrived at 1394046094 with bad export cookie 16896125447898920580 Mar 5 14:01:09 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0015_UUID' is not available for connect (no target) Mar 5 14:01:09 dc2oss04 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 14:02:02 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0024_UUID' is not available for connect (no target) Mar 5 14:02:02 dc2oss03 kernel: : LustreError: Skipped 28 previous similar messages Mar 5 14:01:46 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0054: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88182894f000 cur 1394046106 last 1394046106 Mar 5 14:01:46 dc2oss09 kernel: : Lustre: 16287:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 106 previous similar messages Mar 5 14:01:49 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181c684400 cur 1394046109 last 1394046109 Mar 5 14:01:49 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 104 previous similar messages Mar 5 14:01:54 dc2oss08 kernel: : Lustre: 5775:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1394046106/real 1394046106] req@ffff880177ba2c00 x1456766190369439/t0(0) o105->dc2-OST0050@149.165.229.58@tcp:15/16 lens 344/192 e 0 to 1 dl 1394046114 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 14:01:54 dc2oss08 kernel: : LustreError: 138-a: dc2-OST0050: A client on nid 149.165.229.58@tcp was evicted due to a lock completion callback time out: rc -107 Mar 5 14:01:50 dc2oss10 kernel: : Lustre: dc2-OST005f: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:01:50 dc2oss10 kernel: : Lustre: Skipped 106 previous similar messages Mar 5 14:02:32 dc2oss13 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0087: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c1018ac00 cur 1394046152 last 1394046152 Mar 5 14:02:32 dc2oss13 kernel: : Lustre: 16135:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 14:02:36 dc2oss15 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0096: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c267d7c00 cur 1394046156 last 1394046156 Mar 5 14:02:36 dc2oss15 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 129 previous similar messages Mar 5 14:02:33 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0026: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t37211680 exp ffff880bf3d03400 cur 1394046153 last 1394046153 Mar 5 14:02:33 dc2oss04 kernel: : Lustre: 16112:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 14:02:41 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0074_UUID' is not available for connect (no target) Mar 5 14:02:41 dc2oss11 kernel: : LustreError: Skipped 27 previous similar messages Mar 5 14:02:58 dc2oss09 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 113s: evicting client at 149.165.229.93@tcp ns: filter-dc2-OST0055_UUID lock: ffff881396371d80/0xacae7902c58e98a0 lrc: 3/0,0 mode: PR/PR res: 23134691/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x43e39ea4315f639b expref: 6 pid: 14563 timeout 9061402615 Mar 5 14:03:07 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST004a_UUID' is not available for connect (no target) Mar 5 14:03:07 dc2oss07 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 14:03:33 dc2oss14 kernel: : Lustre: 15768:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bc521e000 cur 1394046213 last 1394046213 Mar 5 14:03:33 dc2oss14 kernel: : Lustre: 15768:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 128 previous similar messages Mar 5 14:03:59 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 574496, left: 574496, last alive is 0 secs ago Mar 5 14:03:59 dc2xfer01 kernel: : LustreError: 2777:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 574496, left: 574496, last alive is 0 secs ago Mar 5 14:04:22 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a3: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818040c2000 cur 1394046262 last 1394046262 Mar 5 14:04:22 dc2oss16 kernel: : Lustre: 15794:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 14:04:53 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c11149c00 cur 1394046293 last 1394046293 Mar 5 14:04:53 dc2oss06 kernel: : Lustre: 15822:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 14:05:26 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0068: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c129de400 cur 1394046326 last 1394046326 Mar 5 14:05:26 dc2oss10 kernel: : Lustre: 16508:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 102 previous similar messages Mar 5 14:05:37 dc2oss02 kernel: : Lustre: dc2-OST000c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:05:37 dc2oss02 kernel: : Lustre: Skipped 99 previous similar messages Mar 5 14:05:37 dc2oss02 kernel: : Lustre: 16491:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c05e93c00 cur 1394046337 last 1394046337 Mar 5 14:05:37 dc2oss02 kernel: : Lustre: 16491:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 95 previous similar messages Mar 5 14:05:26 dc2oss05 kernel: : Lustre: 15973:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c017b7400 cur 1394046326 last 1394046326 Mar 5 14:05:26 dc2oss05 kernel: : Lustre: 15973:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 121 previous similar messages Mar 5 14:05:31 dc2oss14 kernel: : LustreError: 137-5: UUID 'dc2-OST0083_UUID' is not available for connect (no target) Mar 5 14:05:31 dc2oss14 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 14:05:26 dc2oss04 kernel: : Lustre: dc2-OST0021: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:05:26 dc2oss04 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 14:05:52 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 107872, left: 107872, last alive is 23 secs ago Mar 5 14:05:52 dc2xfer04 kernel: : LustreError: 2705:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:1021, with error, wanted: 107872, left: 107872, last alive is 23 secs ago Mar 5 14:06:20 dc2oss03 kernel: : Lustre: dc2-OST0016: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:20 dc2oss03 kernel: : Lustre: Skipped 109 previous similar messages Mar 5 14:06:20 dc2oss03 kernel: : Lustre: 17035:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t31136294 exp ffff880c0266f000 cur 1394046380 last 1394046380 Mar 5 14:06:20 dc2oss03 kernel: : Lustre: 17035:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages Mar 5 14:06:04 dc2oss14 kernel: : Lustre: dc2-OST008e: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:04 dc2oss14 kernel: : Lustre: Skipped 139 previous similar messages Mar 5 14:06:18 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 296384, left: 296384, last alive is 39 secs ago Mar 5 14:06:18 dc2xfer02 kernel: : LustreError: 2697:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 296384, left: 296384, last alive is 39 secs ago Mar 5 14:06:16 dc2oss16 kernel: : Lustre: dc2-OST00a2: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:16 dc2oss16 kernel: : Lustre: Skipped 115 previous similar messages Mar 5 14:06:44 dc2oss11 kernel: : Lustre: 15496:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006e: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817f7ad9c00 cur 1394046404 last 1394046404 Mar 5 14:06:44 dc2oss11 kernel: : Lustre: 15496:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 91 previous similar messages Mar 5 14:06:44 dc2oss16 kernel: : LustreError: 137-5: UUID 'dc2-OST009b_UUID' is not available for connect (no target) Mar 5 14:06:44 dc2oss16 kernel: : LustreError: Skipped 35 previous similar messages Mar 5 14:06:55 dc2oss13 kernel: : Lustre: dc2-OST0086: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:55 dc2oss13 kernel: : Lustre: Skipped 113 previous similar messages Mar 5 14:06:56 dc2oss09 kernel: : Lustre: dc2-OST0055: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:56 dc2oss09 kernel: : Lustre: Skipped 112 previous similar messages Mar 5 14:07:12 dc2oss02 kernel: : LustreError: 137-5: UUID 'dc2-OST000a_UUID' is not available for connect (no target) Mar 5 14:07:12 dc2oss02 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 14:06:53 dc2oss01 kernel: : Lustre: dc2-OST0003: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:06:53 dc2oss01 kernel: : Lustre: Skipped 105 previous similar messages Mar 5 14:06:53 dc2oss01 kernel: : Lustre: 15656:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880beac8ac00 cur 1394046413 last 1394046413 Mar 5 14:06:53 dc2oss01 kernel: : Lustre: 15656:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 105 previous similar messages Mar 5 14:07:15 dc2oss08 kernel: : LustreError: 137-5: UUID 'dc2-OST003f_UUID' is not available for connect (no target) Mar 5 14:07:15 dc2oss08 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 14:07:12 dc2oss01 kernel: : LustreError: 137-5: UUID 'dc2-OST0014_UUID' is not available for connect (no target) Mar 5 14:07:12 dc2oss01 kernel: : LustreError: Skipped 24 previous similar messages Mar 5 14:07:32 dc2oss15 kernel: : Lustre: dc2-OST009c: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:07:32 dc2oss15 kernel: : Lustre: Skipped 134 previous similar messages Mar 5 14:07:40 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST0039_UUID' is not available for connect (no target) Mar 5 14:07:40 dc2oss05 kernel: : LustreError: Skipped 26 previous similar messages Mar 5 14:07:45 dc2oss15 kernel: : LustreError: 137-5: UUID 'dc2-OST00a7_UUID' is not available for connect (no target) Mar 5 14:07:45 dc2oss15 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 14:07:51 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST008e_UUID' is not available for connect (no target) Mar 5 14:07:51 dc2oss13 kernel: : LustreError: Skipped 38 previous similar messages Mar 5 14:07:44 dc2mds01 kernel: : Lustre: MGS: Client aed26e38-a80e-b2f8-7aca-9893fc2aadb1 (at 129.79.10.14@tcp) reconnecting Mar 5 14:07:44 dc2mds01 kernel: : Lustre: Skipped 33 previous similar messages Mar 5 14:07:44 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from aed26e38-a80e-b2f8-7aca-9893fc2aadb1@129.79.10.14@tcp t0 exp ffff883054420800 cur 1394046464 last 1394046464 Mar 5 14:07:44 dc2mds01 kernel: : Lustre: 2352:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 33 previous similar messages Mar 5 14:08:05 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8817d1a0f400 cur 1394046485 last 1394046485 Mar 5 14:08:05 dc2oss08 kernel: : Lustre: 14458:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 80 previous similar messages Mar 5 14:08:18 dc2oss11 kernel: : Lustre: Service thread pid 14647 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 14:08:18 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 14:08:18 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394046498.14647 Mar 5 14:08:44 dc2oss11 kernel: : Lustre: Service thread pid 25399 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 14:08:44 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 14:08:44 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394046524.25399 Mar 5 14:08:45 dc2oss11 kernel: : Lustre: Service thread pid 8707 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 14:08:45 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 14:08:45 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394046525.8707 Mar 5 14:08:49 dc2oss11 kernel: : Lustre: Service thread pid 8789 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 5 14:08:49 dc2oss11 kernel: : [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Mar 5 14:08:49 dc2oss11 kernel: : LustreError: dumping log to /tmp/lustre-log.1394046529.8789 Mar 5 14:09:01 dc2oss06 kernel: : Lustre: dc2-OST003b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:09:01 dc2oss06 kernel: : Lustre: Skipped 95 previous similar messages Mar 5 14:09:16 dc2oss12 kernel: : Lustre: dc2-OST0079: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:09:16 dc2oss12 kernel: : Lustre: Skipped 103 previous similar messages Mar 5 14:09:06 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation Mar 5 14:09:06 dc2mds01 kernel: : LustreError: 17953:0:(obd_class.h:503:obd_set_info_async()) Skipped 23 previous similar messages Mar 5 14:09:35 dc2oss05 kernel: : Lustre: dc2-OST0030: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:09:35 dc2oss05 kernel: : Lustre: Skipped 120 previous similar messages Mar 5 14:09:47 dc2oss11 kernel: : Lustre: dc2-OST006d: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:09:47 dc2oss11 kernel: : Lustre: Skipped 84 previous similar messages Mar 5 14:10:19 dc2oss06 kernel: : LustreError: 137-5: UUID 'dc2-OST002e_UUID' is not available for connect (no target) Mar 5 14:10:19 dc2oss06 kernel: : LustreError: Skipped 25 previous similar messages Mar 5 14:10:36 dc2oss07 kernel: : Lustre: dc2-OST0040: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:10:36 dc2oss07 kernel: : Lustre: Skipped 111 previous similar messages Mar 5 14:10:36 dc2oss07 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff8818160fe000 cur 1394046636 last 1394046636 Mar 5 14:10:36 dc2oss07 kernel: : Lustre: 16129:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 111 previous similar messages Mar 5 14:10:56 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 18272, left: 18272, last alive is 46 secs ago Mar 5 14:10:56 dc2xfer03 kernel: : LustreError: 2639:0:(socklnd.c:1674:ksocknal_destroy_conn()) Completing partial receive from 12345-149.165.235.18@tcp[2], ip 149.165.235.18:988, with error, wanted: 18272, left: 18272, last alive is 46 secs ago Mar 5 14:10:58 dc2oss08 kernel: : Lustre: dc2-OST004b: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:10:58 dc2oss08 kernel: : Lustre: Skipped 81 previous similar messages Mar 5 14:11:01 149.165.238.1 kernel: kernel: [4321033.924047] Lustre: 58791:0:(client.c:1492:ptlrpc_expire_one_request()) @@@ Request x1461669106487812 sent from dc2-OST005b-osc-ffff880956851000 to NID 10.10.0.9@o2ib 7s ago has timed out (7s prior to deadline). Mar 5 14:11:01 149.165.238.1 kernel: kernel: [4321033.924058] Lustre: dc2-OST005b-osc-ffff880956851000: Connection to service dc2-OST005b via nid 10.10.0.9@o2ib was lost; in progress operations using this service will wait for recovery to complete. Mar 5 14:11:01 149.165.238.1 kernel: kernel: [4321033.924405] Lustre: dc2-OST005b-osc-ffff880956851000: Connection restored to service dc2-OST005b using nid 10.10.0.9@o2ib. Mar 5 14:11:01 149.165.238.1 kernel: kernel: [4321033.924423] Lustre: Server dc2-OST005b_UUID version (2.1.6.0) is much newer than client version (1.8.6) Mar 5 14:11:06 dc2oss10 kernel: : LustreError: 137-5: UUID 'dc2-OST005a_UUID' is not available for connect (no target) Mar 5 14:11:06 dc2oss10 kernel: : LustreError: Skipped 31 previous similar messages Mar 5 14:11:21 dc2oss09 kernel: : LustreError: 137-5: UUID 'dc2-OST0066_UUID' is not available for connect (no target) Mar 5 14:11:21 dc2oss09 kernel: : LustreError: Skipped 19 previous similar messages Mar 5 14:11:10 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202f0f961:0x3a17:0x0]] [valid 68719476736] Mar 5 14:11:10 dc2mds01 kernel: : Lustre: 11140:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 21 previous similar messages Mar 5 14:11:49 dc2oss12 kernel: : LustreError: 137-5: UUID 'dc2-OST006f_UUID' is not available for connect (no target) Mar 5 14:11:49 dc2oss12 kernel: : LustreError: Skipped 20 previous similar messages Mar 5 14:12:17 dc2oss03 kernel: : LustreError: 137-5: UUID 'dc2-OST0022_UUID' is not available for connect (no target) Mar 5 14:12:17 dc2oss03 kernel: : LustreError: Skipped 23 previous similar messages Mar 5 14:11:54 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0058: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c2640a400 cur 1394046714 last 1394046714 Mar 5 14:11:54 dc2oss09 kernel: : Lustre: 14513:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 126 previous similar messages Mar 5 14:12:00 dc2oss10 kernel: : Lustre: dc2-OST0063: Client adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5 (at 129.79.10.14@tcp) reconnecting Mar 5 14:12:00 dc2oss10 kernel: : Lustre: Skipped 90 previous similar messages Mar 5 14:12:04 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bf9211c00 cur 1394046724 last 1394046724 Mar 5 14:12:04 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages Mar 5 14:12:15 dc2oss14 kernel: : Lustre: 14592:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1394046727/real 0] req@ffff8813d99cf000 x1456766178706205/t0(0) o104->dc2-OST008d@149.165.226.205@tcp:15/16 lens 296/192 e 0 to 1 dl 1394046735 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 5 14:12:15 dc2oss14 kernel: : LustreError: 138-a: dc2-OST008d: A client on nid 149.165.226.205@tcp was evicted due to a lock blocking callback time out: rc -107 Mar 5 14:12:25 dc2oss04 kernel: : LustreError: 137-5: UUID 'dc2-OST0016_UUID' is not available for connect (no target) Mar 5 14:12:25 dc2oss04 kernel: : LustreError: Skipped 32 previous similar messages Mar 5 14:12:32 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c111f7c00 cur 1394046752 last 1394046752 Mar 5 14:12:32 dc2oss13 kernel: : Lustre: 15935:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 114 previous similar messages Mar 5 14:12:35 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0021: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880c004d5800 cur 1394046755 last 1394046755 Mar 5 14:12:35 dc2oss04 kernel: : Lustre: 16668:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 87 previous similar messages Mar 5 14:12:46 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009b: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff88181e5d5000 cur 1394046766 last 1394046766 Mar 5 14:12:46 dc2oss15 kernel: : Lustre: 14515:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 116 previous similar messages Mar 5 14:13:13 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST007b_UUID' is not available for connect (no target) Mar 5 14:13:13 dc2oss11 kernel: : LustreError: Skipped 21 previous similar messages Mar 5 14:13:16 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0050_UUID' is not available for connect (no target) Mar 5 14:13:16 dc2oss07 kernel: : LustreError: Skipped 16 previous similar messages Mar 5 14:13:33 dc2oss14 kernel: : Lustre: 15701:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from adc9bfa5-dcf7-5426-7ed1-a1ddfa428cf5@129.79.10.14@tcp t0 exp ffff880bef34d800 cur 1394046813 last 1394046813 Mar 5 14:13:33 dc2oss14 kernel: : Lustre: 15701:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 98 previous similar messages