Jul 16 14:30:11 atlas-oss3b5.ccs.ornl.gov kernel: [1820143.452717] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.224@o2ib rejected: o2iblnd fatal error
Jul 16 14:30:11 atlas-oss3b5.ccs.ornl.gov kernel: [1820143.479149] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 5 previous similar messages
Jul 16 14:40:54 atlas-oss3b5.ccs.ornl.gov kernel: [1820786.548200] LNetError: 3041:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 14:43:24 atlas-oss3b5.ccs.ornl.gov kernel: [1820937.276426] LNetError: 3042:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 14:43:24 atlas-oss3b5.ccs.ornl.gov kernel: [1820937.308263] LNetError: 3042:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 2 previous similar messages
Jul 16 14:46:00 atlas-oss3b5.ccs.ornl.gov kernel: [1821092.717191] Lustre: atlas2-OST02dc: Client 96d1bb1c-59dc-a499-e8db-eb8dacc46e30 (at 10.38.144.169@o2ib4) reconnecting
Jul 16 14:48:26 atlas-oss3b5.ccs.ornl.gov kernel: [1821238.934317] LNetError: 3041:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 14:48:26 atlas-oss3b5.ccs.ornl.gov kernel: [1821238.962876] LNetError: 3041:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 5 previous similar messages
Jul 16 14:53:11 atlas-oss3b5.ccs.ornl.gov kernel: [1821523.820322] Lustre: atlas2-OST036c: Client bbc5f8db-891f-3f48-0979-66693b4f37af (at 10.38.144.11@o2ib4) reconnecting
Jul 16 14:53:45 atlas-oss3b5.ccs.ornl.gov kernel: [1821558.205845] Lustre: atlas2-OST012c: Client 810e4a97-0566-03a2-3bb0-d403a8081342 (at 10.38.146.46@o2ib4) reconnecting
Jul 16 14:58:29 atlas-oss3b5.ccs.ornl.gov kernel: [1821841.983660] LNetError: 3041:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 14:58:29 atlas-oss3b5.ccs.ornl.gov kernel: [1821842.010672] LNetError: 3041:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:08:31 atlas-oss3b5.ccs.ornl.gov kernel: [1822444.565156] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:08:31 atlas-oss3b5.ccs.ornl.gov kernel: [1822444.588642] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:18:33 atlas-oss3b5.ccs.ornl.gov kernel: [1823047.060241] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:18:33 atlas-oss3b5.ccs.ornl.gov kernel: [1823047.085829] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:28:35 atlas-oss3b5.ccs.ornl.gov kernel: [1823649.554036] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:28:35 atlas-oss3b5.ccs.ornl.gov kernel: [1823649.583884] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:38:38 atlas-oss3b5.ccs.ornl.gov kernel: [1824252.049700] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:38:38 atlas-oss3b5.ccs.ornl.gov kernel: [1824252.081357] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:48:40 atlas-oss3b5.ccs.ornl.gov kernel: [1824854.544920] LNetError: 3042:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:48:40 atlas-oss3b5.ccs.ornl.gov kernel: [1824854.568963] LNetError: 3042:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 15:58:42 atlas-oss3b5.ccs.ornl.gov kernel: [1825457.039563] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) 10.36.145.229@o2ib rejected: o2iblnd fatal error
Jul 16 15:58:42 atlas-oss3b5.ccs.ornl.gov kernel: [1825457.066667] LNetError: 3035:0:(o2iblnd_cb.c:2635:kiblnd_rejected()) Skipped 11 previous similar messages
Jul 16 16:10:55 atlas-oss3b5.ccs.ornl.gov kernel: [1826189.579403] Lustre: atlas2-OST036c: haven't heard from client bd168378-14ed-c747-2287-316720d642c5 (at 3792@gni103) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881048409c00, cur 1405541455 expire 1405540555 last 1405540103
Jul 16 16:10:55 atlas-oss3b5.ccs.ornl.gov kernel: [1826189.643103] Lustre: Skipped 3 previous similar messages
Jul 16 23:06:38 atlas-oss3b5.ccs.ornl.gov kernel: [1851142.732302] Lustre: atlas2-OST036c: haven't heard from client 455248d5-e7ec-06d7-8fb4-ee74a1df9512 (at 10.36.205.207@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88102c52dc00, cur 1405566398 expire 1405565498 last 1405565045
Jul 16 23:06:38 atlas-oss3b5.ccs.ornl.gov kernel: [1851142.802120] Lustre: Skipped 6 previous similar messages
Jul 16 23:06:42 atlas-oss3b5.ccs.ornl.gov kernel: [1851146.081028] Lustre: atlas2-OST000c: haven't heard from client 455248d5-e7ec-06d7-8fb4-ee74a1df9512 (at 10.36.205.207@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880e59282800, cur 1405566402 expire 1405565502 last 1405565050
Jul 16 23:06:43 atlas-oss3b5.ccs.ornl.gov kernel: [1851147.483008] Lustre: atlas2-OST009c: haven't heard from client 455248d5-e7ec-06d7-8fb4-ee74a1df9512 (at 10.36.205.207@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880f7864d400, cur 1405566403 expire 1405565503 last 1405565051
Jul 16 23:06:45 atlas-oss3b5.ccs.ornl.gov kernel: [1851148.999275] Lustre: atlas2-OST02dc: haven't heard from client 455248d5-e7ec-06d7-8fb4-ee74a1df9512 (at 10.36.205.207@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88101c4a3400, cur 1405566405 expire 1405565505 last 1405565053
Jul 16 23:06:45 atlas-oss3b5.ccs.ornl.gov kernel: [1851149.064344] Lustre: Skipped 3 previous similar messages
Jul 17 07:28:17 atlas-oss3b5.ccs.ornl.gov kernel: [1881252.286572] Lustre: atlas2-OST024c: haven't heard from client 5811d005-4b24-0292-01bf-a21a1f2e1488 (at 14843@gni107) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880d6b9a8c00, cur 1405596497 expire 1405595597 last 1405595145
Jul 17 13:13:16 atlas-oss3b5.ccs.ornl.gov kernel: [1901959.072434] Lustre: atlas2-OST024c: haven't heard from client 4473cdce-f6a8-c605-ff87-c9d0898eff9d (at 17845@gni110) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881028327c00, cur 1405617196 expire 1405616296 last 1405615844
Jul 17 13:13:16 atlas-oss3b5.ccs.ornl.gov kernel: [1901959.140386] Lustre: Skipped 6 previous similar messages
Jul 17 13:13:16 atlas-oss3b5.ccs.ornl.gov kernel: [1901959.160082] Lustre: atlas2-OST024c: haven't heard from client 46646f65-dfd1-917a-69a6-995549ea1ee9 (at 17489@gni110) in 1351 seconds. I think it's dead, and I am evicting it. exp ffff880b2d682400, cur 1405617196 expire 1405616296 last 1405615845
Jul 17 14:16:46 atlas-oss3b5.ccs.ornl.gov kernel: [1905771.092476] Lustre: atlas2-OST01bc: Client 810e4a97-0566-03a2-3bb0-d403a8081342 (at 10.38.146.46@o2ib4) reconnecting
Jul 17 15:32:45 atlas-oss3b5.ccs.ornl.gov kernel: [1910331.772168] LustreError: 69966:0:(ldlm_resource.c:1165:ldlm_resource_get()) atlas2-OST024c: lvbo_init failed for resource 0x216306:0x0: rc = -2
Jul 17 15:32:45 atlas-oss3b5.ccs.ornl.gov kernel: [1910331.804660] LustreError: 69966:0:(ldlm_resource.c:1165:ldlm_resource_get()) Skipped 1 previous similar message
Jul 17 16:16:13 atlas-oss3b5.ccs.ornl.gov kernel: [1912940.346483] LustreError: 12458:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.771949] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329]  req@ffff880416cf9400 x1471904514716844/t0(0) o400->atlas2-MDT0000-lwp-OST02dc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.771965] Lustre: atlas2-MDT0000-lwp-OST01bc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.920802] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.951370] Lustre: atlas2-MDT0000-lwp-OST02dc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914665.010844] Lustre: Skipped 4 previous similar messages
Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914665.021964] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329]  req@ffff880416cf9000 x1471904514716840/t0(0) o400->atlas2-MDT0000-lwp-OST024c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1914814.828359] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479]  req@ffff8802ad30f800 x1471904514716860/t0(0) o400->atlas2-MDT0000-lwp-OST009c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1914814.918669] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jul 17 16:49:36 atlas-oss3b5.ccs.ornl.gov kernel: [1914944.226023] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896]  req@ffff88063e4e4c00 x1471904514716964/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:49:36 atlas-oss3b5.ccs.ornl.gov kernel: [1914944.317480] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jul 17 16:49:56 atlas-oss3b5.ccs.ornl.gov kernel: [1914964.233538] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629]  req@ffff880026decc00 x1471904514716904/t0(0) o400->atlas2-MDT0000-lwp-OST024c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:49:56 atlas-oss3b5.ccs.ornl.gov kernel: [1914964.335027] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 16:52:26 atlas-oss3b5.ccs.ornl.gov kernel: [1915114.289940] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779]  req@ffff880692395000 x1471904514716944/t0(0) o400->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:52:26 atlas-oss3b5.ccs.ornl.gov kernel: [1915114.381257] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 16:54:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915245.339217] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197]  req@ffff88081165e000 x1471904514717008/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:54:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915245.430947] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 16:56:39 atlas-oss3b5.ccs.ornl.gov kernel: [1915367.462818] Lustre: atlas2-OST009c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870859c00, cur 1405630599 expire 1405629699 last 1405629246
Jul 17 16:56:39 atlas-oss3b5.ccs.ornl.gov kernel: [1915367.526463] Lustre: Skipped 19 previous similar messages
Jul 17 16:56:40 atlas-oss3b5.ccs.ornl.gov kernel: [1915368.643499] Lustre: atlas2-OST012c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805484d5800, cur 1405630600 expire 1405629700 last 1405629248
Jul 17 16:56:41 atlas-oss3b5.ccs.ornl.gov kernel: [1915369.902225] Lustre: atlas2-OST01bc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880301a65400, cur 1405630601 expire 1405629701 last 1405629249
Jul 17 16:56:42 atlas-oss3b5.ccs.ornl.gov kernel: [1915370.945701] Lustre: atlas2-OST036c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806f665d800, cur 1405630602 expire 1405629702 last 1405629248
Jul 17 16:56:42 atlas-oss3b5.ccs.ornl.gov kernel: [1915371.009156] Lustre: Skipped 3 previous similar messages
Jul 17 16:59:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915545.452098] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497]  req@ffff8803142adc00 x1471904514717044/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 16:59:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915545.544406] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 17:04:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915845.564889] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797]  req@ffff8808552e2400 x1471904514717080/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:04:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915845.657745] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 17:06:53 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.212253] LNetError: 25848:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds
Jul 17 17:06:53 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.239140] LNetError: 25848:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63
Jul 17 17:06:54 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.279915] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213]  req@ffff8806d9c73000 x1471904514717116/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:06:54 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.379228] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jul 17 17:09:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916115.695488] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347]  req@ffff8803f2266800 x1471904514717124/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:14:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916415.808375] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647]  req@ffff8805522d2c00 x1471904514717164/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:14:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916415.893058] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jul 17 17:24:07 atlas-oss3b5.ccs.ornl.gov kernel: [1917016.034087] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247]  req@ffff88023f372000 x1471904514717244/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:24:07 atlas-oss3b5.ccs.ornl.gov kernel: [1917016.120231] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Jul 17 17:36:37 atlas-oss3b5.ccs.ornl.gov kernel: [1917766.316157] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997]  req@ffff88034bd8d400 x1471904514717332/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:36:37 atlas-oss3b5.ccs.ornl.gov kernel: [1917766.403537] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
Jul 17 17:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1918416.565657] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647]  req@ffff8802bfa1d000 x1471904514717444/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jul 17 17:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1918416.658640] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.701474] LustreError: 167-0: atlas2-MDT0000-lwp-OST036c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail.
Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.739561] LustreError: 167-0: atlas2-MDT0000-lwp-OST02dc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail.
Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.740153] Lustre: atlas2-MDT0000-lwp-OST036c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib)
Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.820232] Lustre: atlas2-MDT0000-lwp-OST02dc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib)
Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.849561] Lustre: Skipped 4 previous similar messages
Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.757892] LustreError: 167-0: atlas2-MDT0000-lwp-OST000c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail.
Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.796225] LustreError: Skipped 4 previous similar messages
Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.816704] Lustre: atlas2-MDT0000-lwp-OST000c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib)
Jul 17 20:39:16 atlas-oss3b5.ccs.ornl.gov kernel: [1928729.460683] Lustre: atlas2-OST000c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting
Jul 17 20:39:16 atlas-oss3b5.ccs.ornl.gov kernel: [1928729.485978] Lustre: Skipped 1 previous similar message
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.099051] Lustre: atlas2-OST000c: deleting orphan objects from 0x0:3068794 to 0x0:3068833
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.099470] Lustre: atlas2-OST009c: deleting orphan objects from 0x0:2461180 to 0x0:2461217
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.099898] Lustre: atlas2-OST012c: deleting orphan objects from 0x0:2301636 to 0x0:2301665
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.101200] Lustre: atlas2-OST01bc: deleting orphan objects from 0x0:2230388 to 0x0:2230433
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.101204] Lustre: atlas2-OST024c: deleting orphan objects from 0x0:2191355 to 0x0:2191425
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.103036] Lustre: atlas2-OST02dc: deleting orphan objects from 0x0:2157952 to 0x0:2157985
Jul 17 23:59:51 atlas-oss3b5.ccs.ornl.gov kernel: [1940769.104425] Lustre: atlas2-OST036c: deleting orphan objects from 0x0:2136202 to 0x0:2136289
Jul 18 10:04:15 atlas-oss3b5.ccs.ornl.gov kernel: [1977046.926426] Lustre: atlas2-OST000c: haven't heard from client fb59aa8b-7f68-9180-a3bb-606a4fe3780f (at 10.36.202.142@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880994b8b800, cur 1405692255 expire 1405691355 last 1405690903
Jul 18 13:27:45 atlas-oss3b5.ccs.ornl.gov kernel: [1989260.968071] Lustre: atlas2-OST01bc: haven't heard from client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804d228f800, cur 1405704465 expire 1405703565 last 1405703113
Jul 18 13:27:45 atlas-oss3b5.ccs.ornl.gov kernel: [1989261.037988] Lustre: Skipped 20 previous similar messages
Jul 18 13:27:47 atlas-oss3b5.ccs.ornl.gov kernel: [1989263.000524] Lustre: atlas2-OST009c: haven't heard from client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880463c00c00, cur 1405704467 expire 1405703567 last 1405703115
Jul 18 13:27:48 atlas-oss3b5.ccs.ornl.gov kernel: [1989263.906522] Lustre: atlas2-OST036c: haven't heard from client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880251cc5400, cur 1405704468 expire 1405703568 last 1405703116
Jul 18 13:27:48 atlas-oss3b5.ccs.ornl.gov kernel: [1989263.969829] Lustre: Skipped 1 previous similar message