Jan 30 10:34:50 sprig2 kernel: [682444.492896] LustreError: 11-0: scratch-OST0002-osc-ffff883f7e76a000: Communicating with 11.3.0.14@o2ib, operation obd_ping failed with -107.
Jan 30 10:34:50 sprig2 kernel: [682444.492905] LustreError: Skipped 1 previous similar message
Jan 30 10:34:50 sprig2 kernel: [682444.492912] Lustre: scratch-OST0000-osc-ffff883f7e76a000: Connection to scratch-OST0000 (at 11.3.0.14@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 30 10:34:50 sprig2 kernel: [682444.492921] Lustre: Skipped 2 previous similar messages
Jan 30 10:34:50 sprig2 kernel: [682444.493271] LustreError: 167-0: scratch-OST0001-osc-ffff883f7e76a000: This client was evicted by scratch-OST0001; in progress operations using this service will fail.
Jan 30 10:34:50 sprig2 kernel: [682444.493529] Lustre: 9546:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001452:0x11cdb:0x0]//gvik/tmp/intelremotemonagent.3008.log may get corrupted (rc -108)
Jan 30 10:34:50 sprig2 kernel: klogd 1.4.1, ---------- state change ----------
Jan 30 10:34:53 sprig2 kernel: [682447.572819] Lustre: scratch-OST0009-osc-ffff883f7e76a000: Connection restored to scratch-OST0009 (at 11.3.0.15@o2ib)
Jan 30 10:34:55 sprig2 kernel: [682449.066670] Lustre: scratch-OST0005-osc-ffff883f7e76a000: Connection restored to scratch-OST0005 (at 11.3.0.14@o2ib)
Jan 30 10:35:15 sprig2 kernel: [682469.451776] LustreError: 11-0: scratch-OST0003-osc-ffff883f7e76a000: Communicating with 11.3.0.14@o2ib, operation obd_ping failed with -107.
Jan 30 10:35:15 sprig2 kernel: [682469.451784] LustreError: Skipped 9 previous similar messages
Jan 30 10:35:15 sprig2 kernel: [682469.451791] Lustre: scratch-OST0004-osc-ffff883f7e76a000: Connection to scratch-OST0004 (at 11.3.0.14@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 30 10:35:15 sprig2 kernel: [682469.451800] Lustre: Skipped 8 previous similar messages
Jan 30 10:35:15 sprig2 kernel: [682469.452178] LustreError: 167-0: scratch-OST0003-osc-ffff883f7e76a000: This client was evicted by scratch-OST0003; in progress operations using this service will fail.
Jan 30 10:35:15 sprig2 kernel: [682469.452186] LustreError: Skipped 9 previous similar messages
Jan 30 10:36:30 sprig2 kernel: [682544.328079] LustreError: 226870:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0006_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:36:30 sprig2 kernel: [682544.328087] LustreError: 226870:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:36:30 sprig2 kernel: [682544.328104] LustreError: 226867:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f94cc00 x1457934633486060/t0(0) o4->scratch-OST000b-osc-ffff883f7e76a000@11.3.0.15@o2ib:6/4 lens 488/448 e 0 to 0 dl 1391077977 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Jan 30 10:36:30 sprig2 kernel: [682544.328110] LustreError: 226862:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0001_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:36:30 sprig2 kernel: [682544.328113] LustreError: 226867:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 3 previous similar messages
Jan 30 10:36:30 sprig2 kernel: [682544.328117] LustreError: 226862:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:36:55 sprig2 kernel: [682569.286948] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0004_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:36:55 sprig2 kernel: [682569.286956] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 4 previous similar messages
Jan 30 10:36:55 sprig2 kernel: [682569.286990] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878400 x1457934556401140/t0(0) o101->scratch-OST0003-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:36:55 sprig2 kernel: [682569.286995] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0004_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:36:55 sprig2 kernel: [682569.286999] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 4 previous similar messages
Jan 30 10:36:55 sprig2 kernel: [682569.287007] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:37:05 sprig2 kernel: [682579.187091] Lustre: 9540:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001487:0x1eaaa:0x0]//prod/ptpm/aldv/1.0/bin/aldv may get corrupted (rc -108)
Jan 30 10:37:05 sprig2 kernel: [682579.187101] Lustre: 9540:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 12 previous similar messages
Jan 30 10:38:10 sprig2 kernel: [682644.163562] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0002_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:38:10 sprig2 kernel: [682644.163573] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:38:10 sprig2 kernel: [682644.163581] LustreError: 226870:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878800 x1457934556401144/t0(0) o101->scratch-OST0006-osc-ffff883f7e76a000@11.3.0.15@o2ib:28/4 lens 328/400 e 0 to 0 dl 1390844424 ref 1 fl Rpc:RE/0/ffffffff rc -5/-1
Jan 30 10:38:10 sprig2 kernel: [682644.163601] LustreError: 226867:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST000b_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:38:10 sprig2 kernel: [682644.163605] LustreError: 226870:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 6 previous similar messages
Jan 30 10:38:10 sprig2 kernel: [682644.163607] LustreError: 226867:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:38:35 sprig2 kernel: [682669.122437] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0004_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:38:35 sprig2 kernel: [682669.122466] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:38:35 sprig2 kernel: [682669.122473] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878400 x1457934556401140/t0(0) o101->scratch-OST0003-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:38:35 sprig2 kernel: [682669.122477] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0004_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:38:35 sprig2 kernel: [682669.122481] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:38:35 sprig2 kernel: [682669.122484] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:39:25 sprig2 kernel: [682718.956452] Lustre: 9535:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001487:0x1eaac:0x0]//prod/ptpm/aldv/1.0/include/aldv.h may get corrupted (rc -108)
Jan 30 10:39:25 sprig2 kernel: [682718.956456] Lustre: 9535:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 8 previous similar messages
Jan 30 10:39:50 sprig2 kernel: [682743.999067] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0002_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:39:50 sprig2 kernel: [682743.999072] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:39:50 sprig2 kernel: [682743.999075] LustreError: 226862:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ce8c73800 x1457934556401156/t0(0) o101->scratch-OST0001-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:39:50 sprig2 kernel: [682743.999079] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0008_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:39:50 sprig2 kernel: [682743.999082] LustreError: 226862:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:39:50 sprig2 kernel: [682743.999084] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:40:01 sprig2 /usr/sbin/cron[226882]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Jan 30 10:40:15 sprig2 kernel: [682768.957917] LustreError: 226878:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0003_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:40:15 sprig2 kernel: [682768.957928] LustreError: 226878:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:40:15 sprig2 kernel: [682768.957945] LustreError: 226879:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878c00 x1457934556401148/t0(0) o101->scratch-OST0004-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:40:15 sprig2 kernel: [682768.957960] LustreError: 226879:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:40:15 sprig2 kernel: [682768.957967] LustreError: 226878:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0003_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:40:15 sprig2 kernel: [682768.957971] LustreError: 226878:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:40:25 sprig2 kernel: [682778.857733] Lustre: 9543:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001487:0x1eaad:0x0]//prod/ptpm/aldv/1.0/include/aldv-mpi.h may get corrupted (rc -108)
Jan 30 10:40:25 sprig2 kernel: [682778.857765] Lustre: 9543:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 28 previous similar messages
Jan 30 10:41:30 sprig2 kernel: [682843.834518] LustreError: 226866:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST000a_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:41:30 sprig2 kernel: [682843.834526] LustreError: 226866:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 5 previous similar messages
Jan 30 10:41:30 sprig2 kernel: [682843.834550] LustreError: 226867:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f94cc00 x1457934633486060/t0(0) o4->scratch-OST000b-osc-ffff883f7e76a000@11.3.0.15@o2ib:6/4 lens 488/448 e 0 to 0 dl 1391077977 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Jan 30 10:41:30 sprig2 kernel: [682843.834564] LustreError: 226868:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0007_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:41:30 sprig2 kernel: [682843.834568] LustreError: 226867:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 7 previous similar messages
Jan 30 10:41:30 sprig2 kernel: [682843.834574] LustreError: 226868:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 4 previous similar messages
Jan 30 10:41:55 sprig2 kernel: [682868.793397] LustreError: 226878:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0003_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:41:55 sprig2 kernel: [682868.793426] LustreError: 226878:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:41:55 sprig2 kernel: [682868.793430] LustreError: 226879:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878c00 x1457934556401148/t0(0) o101->scratch-OST0004-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:41:55 sprig2 kernel: [682868.793435] LustreError: 226878:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0003_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:41:55 sprig2 kernel: [682868.793439] LustreError: 226879:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:41:55 sprig2 kernel: [682868.793443] LustreError: 226878:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Jan 30 10:42:45 sprig2 kernel: [682918.711456] LustreError: 11-0: scratch-OST0005-osc-ffff883f7e76a000: Communicating with 11.3.0.14@o2ib, operation obd_ping failed with -107.
Jan 30 10:42:45 sprig2 kernel: [682918.711476] Lustre: scratch-OST0005-osc-ffff883f7e76a000: Connection to scratch-OST0005 (at 11.3.0.14@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 30 10:42:45 sprig2 kernel: [682918.711873] LustreError: 167-0: scratch-OST0005-osc-ffff883f7e76a000: This client was evicted by scratch-OST0005; in progress operations using this service will fail.
Jan 30 10:42:45 sprig2 kernel: [682918.711880] LustreError: Skipped 1 previous similar message
Jan 30 10:42:45 sprig2 kernel: [682918.712608] LustreError: 226935:0:(osc_lock.c:817:osc_ldlm_completion_ast()) lock@ffff883e7c31e7b8[2 3 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100050000:0xe83b:0x0] {
Jan 30 10:42:45 sprig2 kernel: [682918.712623] LustreError: 226935:0:(osc_lock.c:817:osc_ldlm_completion_ast())     lovsub@ffff883d181bb620: [3 ffff8838837fb818 P(0):[0, 18446744073709551615]@[0x50000040b:0x1500:0x0]] 
Jan 30 10:42:45 sprig2 kernel: [682918.712637] LustreError: 226935:0:(osc_lock.c:817:osc_ldlm_completion_ast())     osc@ffff883e87c3a740: ffff883c2d361b40    0x20000001001 0xcaa0c500ab0aa2a 3 ffff883e6b5d8200 size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Jan 30 10:42:45 sprig2 kernel: [682918.712644] LustreError: 226935:0:(osc_lock.c:817:osc_ldlm_completion_ast()) } lock@ffff883e7c31e7b8
Jan 30 10:42:45 sprig2 kernel: [682918.712650] LustreError: 226935:0:(osc_lock.c:817:osc_ldlm_completion_ast()) dlmlock returned -5
Jan 30 10:42:45 sprig2 kernel: [682918.712752] LustreError: 226935:0:(ldlm_resource.c:804:ldlm_resource_complain()) scratch-OST0005-osc-ffff883f7e76a000: namespace resource [0x2e3695:0x0:0x0].0 (ffff883c6631d5c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Jan 30 10:42:45 sprig2 kernel: [682918.712762] LustreError: 226935:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x2e3695:0x0:0x0].0 (ffff883c6631d5c0) refcount = 2
Jan 30 10:42:45 sprig2 kernel: [682918.712785] LustreError: 226935:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0xe83b:0x0:0x0].0 (ffff883c54dd1940) refcount = 2
Jan 30 10:42:45 sprig2 kernel: [682918.712791] LustreError: 226935:0:(ldlm_resource.c:1418:ldlm_resource_dump()) Granted locks (in reverse order):
Jan 30 10:42:45 sprig2 kernel: [682918.712806] LustreError: 226935:0:(ldlm_resource.c:1421:ldlm_resource_dump()) ### ### ns: scratch-OST0005-osc-ffff883f7e76a000 lock: ffff883c2d361b40/0xcaa0c500ab0aa2a lrc: 3/1,0 mode: PR/PR res: [0xe83b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x126400000000 nid: local remote: 0x7385d82222d550a4 expref: -99 pid: 226893 timeout: 0 lvb_type: 1
Jan 30 10:42:45 sprig2 kernel: [682918.712853] Lustre: scratch-OST0005-osc-ffff883f7e76a000: Connection restored to scratch-OST0005 (at 11.3.0.14@o2ib)
Jan 30 10:42:50 sprig2 intelremotemond[212476]: Consumer: timeout, exiting...
Jan 30 10:42:50 sprig2 intelremotemond[226937]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent
Jan 30 10:42:50 sprig2 intelremotemond[226937]: SendDataToBackend: start agent
Jan 30 10:42:50 sprig2 intelremotemond[212476]: Main: try to delete semaphore, status 0
Jan 30 10:42:50 sprig2 intelremotemond[212476]: Main: daemon stopped, force is 0
Jan 30 10:42:50 sprig2 kernel: [682923.678972] LustreError: 226861:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881ec191c480@{[0 -> 8/255], [3|0|+|rpc|wihY|ffff881f610d8648], [36864|9|+|-|ffff881d1edf9a58|256|ffff8817b35f0300]} scratch-OST000b-osc-ffff883f7e76a000: wait ext to 0 timedout, recovery in progress?
Jan 30 10:43:10 sprig2 kernel: [682943.670000] LustreError: 226867:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST000b_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:43:10 sprig2 kernel: [682943.670008] LustreError: 226867:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Jan 30 10:43:10 sprig2 kernel: [682943.670023] LustreError: 226863:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4faa6c00 x1457934556401164/t0(0) o4->scratch-OST0000-osc-ffff883f7e76a000@11.3.0.14@o2ib:6/4 lens 488/448 e 0 to 0 dl 1390844424 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Jan 30 10:43:10 sprig2 kernel: [682943.670043] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0008_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:43:10 sprig2 kernel: [682943.670047] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 7 previous similar messages
Jan 30 10:43:10 sprig2 kernel: [682943.670050] LustreError: 226863:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 7 previous similar messages
Jan 30 10:43:55 sprig2 kernel: [682988.512517] Lustre: 9544:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001487:0x1eac0:0x0]//prod/ptpm/aldv/1.0/mpt-2.09/lib/libaldv-fortran.so may get corrupted (rc -108)
Jan 30 10:43:55 sprig2 kernel: [682988.512527] Lustre: 9544:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 11 previous similar messages
Jan 30 10:44:15 sprig2 kernel: [683008.479668] Lustre: 9549:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x500001487:0x1eabd:0x0]//prod/ptpm/aldv/1.0/mpt-2.09/bin/aldvtest-mpi may get corrupted (rc -108)
Jan 30 10:44:15 sprig2 kernel: [683008.479687] Lustre: 9549:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 1 previous similar message
Jan 30 10:44:50 sprig2 kernel: [683043.505483] LustreError: 226867:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST000b_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:44:50 sprig2 kernel: [683043.505502] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0002_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:44:50 sprig2 kernel: [683043.505507] LustreError: 226866:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883cf90b5c00 x1457934556401160/t0(0) o101->scratch-OST000a-osc-ffff883f7e76a000@11.3.0.15@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:44:50 sprig2 kernel: [683043.505512] LustreError: 226868:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0007_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:44:50 sprig2 kernel: [683043.505517] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0008_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:44:50 sprig2 kernel: [683043.505523] LustreError: 226864:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 12 previous similar messages
Jan 30 10:44:50 sprig2 kernel: [683043.505527] LustreError: 226866:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 7 previous similar messages
Jan 30 10:44:50 sprig2 kernel: [683043.505531] LustreError: 226868:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 6 previous similar messages
Jan 30 10:44:50 sprig2 kernel: [683043.505535] LustreError: 226871:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 6 previous similar messages
Jan 30 10:46:01 sprig2 syslog-ng[2042]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=950', processed='center(received)=630', processed='destination(messages)=630', processed='destination(mailinfo)=0', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(newserr)=0', processed='destination(mailerr)=0', processed='destination(netmgm)=0', processed='destination(warn)=120', processed='destination(console)=100', processed='destination(null)=0', processed='destination(mail)=0', processed='destination(xconsole)=100', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=630'
Jan 30 10:47:05 sprig2 kernel: [683178.199549] LustreError: 9585:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881ea7b02918@{[0 -> 6/255], [3|0|+|rpc|wihY|ffff881e9d8d8e68], [28672|7|+|-|ffff881d18e87bd8|256|ffff883f7cce2080]} scratch-OST0006-osc-ffff883f7e76a000: wait ext to 0 timedout, recovery in progress?
Jan 30 10:48:10 sprig2 kernel: [683243.176503] LustreError: 226862:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ce8c73800 x1457934556401156/t0(0) o101->scratch-OST0001-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:48:10 sprig2 kernel: [683243.176509] LustreError: 226866:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST000a_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:48:10 sprig2 kernel: [683243.176514] LustreError: 226866:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 18 previous similar messages
Jan 30 10:48:10 sprig2 kernel: [683243.176518] LustreError: 226862:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 23 previous similar messages
Jan 30 10:49:25 sprig2 kernel: [683317.969224] LustreError: 10924:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881f4f4b8a68@{[0 -> 11/255], [3|0|+|rpc|wihY|ffff881d1e6ab4e8], [49152|12|+|-|ffff881d1f259cf8|256|ffff883f78ed2200]} scratch-OST0000-osc-ffff883f7e76a000: wait ext to 0 timedout, recovery in progress?
Jan 30 10:49:50 sprig2 kernel: [683343.011959] LustreError: 226868:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0007_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:49:50 sprig2 kernel: [683343.011963] LustreError: 226868:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 26 previous similar messages
Jan 30 10:53:10 sprig2 kernel: [683542.682934] LustreError: 226866:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883cf90b5c00 x1457934556401160/t0(0) o101->scratch-OST000a-osc-ffff883f7e76a000@11.3.0.15@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 10:53:10 sprig2 kernel: [683542.682940] LustreError: 226864:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0002_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 10:53:10 sprig2 kernel: [683542.682946] LustreError: 226864:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 25 previous similar messages
Jan 30 10:53:10 sprig2 kernel: [683542.682948] LustreError: 226866:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 27 previous similar messages
Jan 30 10:58:35 sprig2 kernel: [683867.148277] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0004_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 10:58:35 sprig2 kernel: [683867.148306] LustreError: 226879:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 51 previous similar messages
Jan 30 11:01:55 sprig2 kernel: [684066.819294] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883d9a878400 x1457934556401140/t0(0) o101->scratch-OST0003-osc-ffff883f7e76a000@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1390844424 ref 1 fl Interpret:RE/0/0 rc -5/0
Jan 30 11:01:55 sprig2 kernel: [684066.819299] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0004_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Jan 30 11:01:55 sprig2 kernel: [684066.819303] LustreError: 226879:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 53 previous similar messages
Jan 30 11:01:55 sprig2 kernel: [684066.819306] LustreError: 226878:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 58 previous similar messages
Jan 30 11:04:25 sprig2 kernel: [684216.572825] LustreError: 11-0: scratch-MDT0000-mdc-ffff883f7e76a000: Communicating with 11.3.0.13@o2ib, operation obd_ping failed with -107.
Jan 30 11:04:25 sprig2 kernel: [684216.572849] Lustre: scratch-MDT0000-mdc-ffff883f7e76a000: Connection to scratch-MDT0000 (at 11.3.0.13@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jan 30 11:04:25 sprig2 kernel: [684216.573224] LustreError: 167-0: scratch-MDT0000-mdc-ffff883f7e76a000: This client was evicted by scratch-MDT0000; in progress operations using this service will fail.
Jan 30 11:04:25 sprig2 kernel: [684216.574378] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) inode 360288055451468550 mdc close failed: rc = -108
Jan 30 11:04:25 sprig2 kernel: [684216.579932] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) inode 360288055451455366 mdc close failed: rc = -108
Jan 30 11:04:25 sprig2 kernel: [684216.631483] LustreError: 227201:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -108
Jan 30 11:04:25 sprig2 kernel: [684216.645795] LustreError: 227201:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -108
Jan 30 11:04:26 sprig2 kernel: [684217.086246] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) inode 360288058723133413 mdc close failed: rc = -108
Jan 30 11:04:26 sprig2 kernel: [684217.086255] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) Skipped 88 previous similar messages
Jan 30 11:04:27 sprig2 kernel: [684218.090918] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) inode 360288058689521293 mdc close failed: rc = -108
Jan 30 11:04:27 sprig2 kernel: [684218.090927] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) Skipped 183 previous similar messages
Jan 30 11:04:29 sprig2 kernel: [684220.093725] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) inode 360288058739893366 mdc close failed: rc = -108
Jan 30 11:04:29 sprig2 kernel: [684220.093735] LustreError: 227200:0:(file.c:163:ll_close_inode_openhandle()) Skipped 339 previous similar messages
Jan 30 11:04:29 sprig2 kernel: [684220.566030] LustreError: 14050:0:(mdc_locks.c:840:mdc_enqueue()) ldlm_cli_enqueue: -108
Jan 30 11:04:29 sprig2 kernel: [684220.566035] LustreError: 14050:0:(mdc_locks.c:840:mdc_enqueue()) Skipped 2 previous similar messages
Jan 30 11:04:29 sprig2 kernel: [684220.566043] LustreError: 14050:0:(vvp_io.c:1230:vvp_io_init()) scratch: refresh file layout [0x50000149b:0xb450:0x0] error -108.
Jan 30 11:04:30 sprig2 kernel: [684221.298635] Lustre: scratch-MDT0000-mdc-ffff883f7e76a000: Connection restored to scratch-MDT0000 (at 11.3.0.13@o2ib)
Jan 30 11:09:50 sprig2 kernel: [684541.037939] LustreError: 226868:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0007_UUID: rc = -110 waiting for callback (1 != 0)
Jan 30 11:09:50 sprig2 kernel: [684541.037949] LustreError: 226868:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 66 previous similar messages