Jan 30 10:34:32 sprig-oss1 kernel: : LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 106s: evicting client at 11.3.0.46@o2ib  ns: filter-scratch-OST0002_UUID lock: ffff8804e1a9a6c0/0x7385d82222d4e22c lrc: 3/0,0 mode: PW/PW res: [0x8b0a5e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab03473 expref: 83001 pid: 31413 timeout: 10851069134 lvb_type: 0
Jan 30 10:34:34 sprig-oss1 kernel: : LustreError: 14634:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_statfs: tot_granted 803686272 != fo_tot_granted 839387008
Jan 30 10:34:34 sprig-oss1 kernel: : LustreError: 14634:0:(ofd_grant.c:169:ofd_grant_sanity_check()) ofd_statfs: tot_dirty 4132864 != fo_tot_dirty 4612096
Jan 30 10:34:35 sprig-oss1 kernel: : LustreError: 32619:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8801ec38a000 x1451777724483484/t0(0) o104->scratch-OST0003@11.3.0.46@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Jan 30 10:34:35 sprig-oss1 kernel: : LustreError: 32619:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Jan 30 10:34:35 sprig-oss1 kernel: : LustreError: 32619:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.3.0.46@o2ib) returned 0 from blocking AST ns: filter-scratch-OST0003_UUID lock: ffff88076fce1480/0x7385d82222d51ea8 lrc: 4/0,0 mode: PW/PW res: [0x8b0af0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab07771 expref: 72883 pid: 31397 timeout: 10851172539 lvb_type: 0
Jan 30 10:34:35 sprig-oss1 kernel: : LustreError: 32619:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.3.0.46@o2ib) returned 0 from blocking AST ns: filter-scratch-OST0003_UUID lock: ffff88057cf0c240/0x7385d82222d51c8d lrc: 4/0,0 mode: PW/PW res: [0x8b0ade:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab06e02 expref: 72803 pid: 4102 timeout: 10851172540 lvb_type: 0
Jan 30 10:34:36 sprig-oss1 kernel: : LustreError: 13976:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_statfs: tot_granted 722820608 != fo_tot_granted 758373888
Jan 30 10:34:36 sprig-oss1 kernel: : LustreError: 13976:0:(ofd_grant.c:169:ofd_grant_sanity_check()) ofd_statfs: tot_dirty 4182016 != fo_tot_dirty 4857856
Jan 30 10:34:39 sprig-oss1 kernel: : LustreError: 14257:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_statfs: tot_granted 709417600 != fo_tot_granted 770718336
Jan 30 10:34:39 sprig-oss1 kernel: : LustreError: 14257:0:(ofd_grant.c:169:ofd_grant_sanity_check()) ofd_statfs: tot_dirty 3620864 != fo_tot_dirty 4136960
Jan 30 10:38:37 sprig-oss1 kernel: : Lustre: scratch-OST0000: haven't heard from client 360e8361-5a5d-012f-2836-620fcd73a7d2 (at 11.3.0.46@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804603fdc00, cur 1391078317 expire 1391078167 last 1391078090
Jan 30 10:38:37 sprig-oss1 kernel: : Lustre: Skipped 4 previous similar messages
Jan 30 10:38:37 sprig-oss1 kernel: : Lustre: scratch-OST0001: haven't heard from client 360e8361-5a5d-012f-2836-620fcd73a7d2 (at 11.3.0.46@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88042ba2b800, cur 1391078317 expire 1391078167 last 1391078090
Jan 30 10:39:03 sprig-oss1 kernel: : Lustre: scratch-OST0004: haven't heard from client 360e8361-5a5d-012f-2836-620fcd73a7d2 (at 11.3.0.46@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff880802785000, cur 1391078343 expire 1391078193 last 1391078115
Jan 30 10:39:03 sprig-oss1 kernel: : Lustre: Skipped 1 previous similar message
Jan 30 10:42:44 sprig-oss1 kernel: : LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 103s: evicting client at 11.3.0.46@o2ib  ns: filter-scratch-OST0005_UUID lock: ffff8805b4324d80/0x7385d82222d5508f lrc: 3/0,0 mode: PR/PR res: [0x8acfeb:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab0a989 expref: 31 pid: 13979 timeout: 10851561118 lvb_type: 1
Jan 30 10:42:44 sprig-oss1 kernel: : LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) Skipped 32 previous similar messages