1373469601 2013 Jul 10 17:20:01 client kern warning kernel Lustre: DEBUG MARKER: Wed Jul 10 17:20:01 2013 1373469601 2013 Jul 10 17:20:01 client kern warning kernel 1373469763 2013 Jul 10 17:22:43 client kern err kernel LustreError: 11-0: an error occurred while communicating with XXX.XXX.XXX.XXX@o2ib. The ost_statfs operation failed with -107 1373469763 2013 Jul 10 17:22:43 client kern warning kernel Lustre: fsname-OST0063-osc-ffff88106cfa5c00: Connection to fsname-OST0063 (at XXX.XXX.XXX.XXX@o2ib) was lost; in progress operations using this service will wait for recovery to complete 1373469763 2013 Jul 10 17:22:43 client kern err kernel LustreError: 167-0: This client was evicted by fsname-OST0063; in progress operations using this service will fail. 1373469898 2013 Jul 10 17:24:58 client kern err kernel LustreError: 61299:0:(client.c:1076:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff881da3851000 x1438818237968438/t0(0) o13->fsname-OST0043-osc-ffff88106cfa5c00@XXX.XXX.XXX.XXX@o2ib:7/4 lens 192/336 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 1373469898 2013 Jul 10 17:24:58 client kern err kernel LustreError: 61299:0:(client.c:1076:ptlrpc_import_delay_req()) Skipped 14 previous similar messages 1373469901 2013 Jul 10 17:25:01 client kern warning kernel Lustre: DEBUG MARKER: Wed Jul 10 17:25:01 2013 1373469901 2013 Jul 10 17:25:01 client kern warning kernel 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 11-0: an error occurred while communicating with XXX.XXX.XXX.XXX@o2ib. The ost_statfs operation failed with -107 1373470063 2013 Jul 10 17:27:43 client kern warning kernel Lustre: fsname-OST0020-osc-ffff88106cfa5c00: Connection to fsname-OST0020 (at XXX.XXX.XXX.XXX@o2ib) was lost; in progress operations using this service will wait for recovery to complete 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 167-0: This client was evicted by fsname-OST0020; in progress operations using this service will fail. 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 62247:0:(osc_lock.c:816:osc_ldlm_completion_ast()) lock@ffff880017f8d278[2 3 0 1 1 00000000] W(2):[0, 18446744073709551615]@[0x100200000:0x29d77f:0x0] { 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 62247:0:(osc_lock.c:816:osc_ldlm_completion_ast()) lovsub@ffff8803358a8320: [0 ffff8802151d9880 W(2):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] [0 ffff881a2712d540 W(2):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] [0 ffff880a936eb5a8 P(0):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 62247:0:(osc_lock.c:816:osc_ldlm_completion_ast()) osc@ffff88038779dd78: ffff880350e4c6c0 80100000 0x8566fb6fad55c51f 3 ffff88005be09bf8 size: 303 mtime: 1373367647 atime: 1367838036 ctime: 1373367647 blocks: 8 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 62247:0:(osc_lock.c:816:osc_ldlm_completion_ast()) } lock@ffff880017f8d278 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 62247:0:(osc_lock.c:816:osc_ldlm_completion_ast()) dlmlock returned -5 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) lock@ffff880e85e16238[3 1 0 0 0 00000005] P(0):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0] { 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) vvp@ffff880dbb85aa10: 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) lov@ffff880a936eb5a8: 4 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) 0 0: --- 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) 1 0: lock@ffff880017f8d0b8[0 5 0 0 0 00000000] W(2):[0, 18446744073709551615]@[0x100010000:0x2ac0cf:0x0] { 1373470063 2013 Jul 10 17:27:43 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) lovsub@ffff8803358a89e0: [1 ffff8802151d9880 W(2):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] [1 ffff880a936eb5a8 P(0):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) osc@ffff88038779dcc0: ffff8805c3c28b40 80100000 0x8566fb6fad55c526 4 (null) size: 0 mtime: 1373367647 atime: 1367836835 ctime: 1373367647 blocks: 0 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) } lock@ffff880017f8d0b8 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) 2 0: lock@ffff880017f8d198[0 5 0 0 0 00000000] W(2):[0, 18446744073709551615]@[0x100530000:0x7915b:0x0] { 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) lovsub@ffff8803358a8260: [2 ffff8802151d9880 W(2):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] [2 ffff880a936eb5a8 P(0):[0, 18446744073709551615]@[0x20db8db6e:0x1db06:0x0]] 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) osc@ffff88038779dc08: ffff8807a2e4f900 80100000 0x8566fb6fad55c52d 4 (null) size: 0 mtime: 1373367647 atime: 1367836817 ctime: 1373367647 blocks: 0 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) } lock@ffff880017f8d198 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) 3 0: --- 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) } lock@ffff880e85e16238 1373470064 2013 Jul 10 17:27:44 client kern err kernel LustreError: 3084:0:(lov_lock.c:785:lov_lock_cancel()) lov_lock_cancel fails with -5. 1373470083 2013 Jul 10 17:28:03 client kern err kernel LustreError: 65257:0:(cl_io.c:1702:cl_sync_io_wait()) SYNC IO failed with error: -110, try to cancel 1 remaining pages 1373470083 2013 Jul 10 17:28:03 client kern err kernel LustreError: 65257:0:(cl_io.c:967:cl_io_cancel()) Canceling ongoing page trasmission