Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 106s: evicting client at 11.3.0.46@o2ib  ns: filter-scratch-OST0009_UUID lock: ffff8804e5c59900/0x82ff997f11762de0 lrc: 3/0,0 mode: PW/PW res: [0x8b0bd4:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab03219 expref: 83003 pid: 16174 timeout: 10507404458 lvb_type: 0
Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 24398:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88046b7cd800 x1452137143493904/t0(0) o104->scratch-OST0006@11.3.0.46@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 24398:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 24398:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.3.0.46@o2ib) returned 0 from blocking AST ns: filter-scratch-OST0006_UUID lock: ffff8801894d1b40/0x82ff997f11766b0b lrc: 4/0,0 mode: PW/PW res: [0x8b0ade:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab076bb expref: 82213 pid: 15699 timeout: 10507505018 lvb_type: 0
Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 24398:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.3.0.46@o2ib) returned 0 from blocking AST ns: filter-scratch-OST0006_UUID lock: ffff88040ad366c0/0x82ff997f1176745e lrc: 4/0,0 mode: PR/PR res: [0x8b0b2b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab0955b expref: 81672 pid: 23470 timeout: 10507505036 lvb_type: 1
Jan 30 10:34:32 sprig-oss2 kernel: : LustreError: 24398:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) Skipped 29 previous similar messages
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_statfs: tot_granted 720381056 != fo_tot_granted 816436352
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:163:ofd_grant_sanity_check()) Skipped 16 previous similar messages
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:169:ofd_grant_sanity_check()) ofd_statfs: tot_dirty 3837952 != fo_tot_dirty 4624384
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:169:ofd_grant_sanity_check()) Skipped 16 previous similar messages
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15696:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff88075cfacc00 x1452137143494056/t0(0) o104->scratch-OST0008@11.3.0.46@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15696:0:(client.c:1049:ptlrpc_import_delay_req()) Skipped 37 previous similar messages
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15696:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.3.0.46@o2ib) returned 0 from blocking AST ns: filter-scratch-OST0008_UUID lock: ffff880728ff6b40/0x82ff997f11766b3c lrc: 4/0,0 mode: PR/PR res: [0x8b0b62:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 11.3.0.46@o2ib remote: 0xcaa0c500ab0776a expref: 76612 pid: 15699 timeout: 10507507019 lvb_type: 1
Jan 30 10:34:34 sprig-oss2 kernel: : LustreError: 15696:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) Skipped 7 previous similar messages
Jan 30 10:34:39 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:163:ofd_grant_sanity_check()) ofd_statfs: tot_granted 862975744 != fo_tot_granted 904980224
Jan 30 10:34:39 sprig-oss2 kernel: : LustreError: 15707:0:(ofd_grant.c:169:ofd_grant_sanity_check()) ofd_statfs: tot_dirty 11501568 != fo_tot_dirty 11649024
Jan 30 10:38:39 sprig-oss2 kernel: : Lustre: scratch-OST0006: haven't heard from client 360e8361-5a5d-012f-2836-620fcd73a7d2 (at 11.3.0.46@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880871304000, cur 1391078319 expire 1391078169 last 1391078090
Jan 30 10:38:39 sprig-oss2 kernel: : Lustre: Skipped 4 previous similar messages
Jan 30 10:38:39 sprig-oss2 kernel: : Lustre: scratch-OST000a: haven't heard from client 360e8361-5a5d-012f-2836-620fcd73a7d2 (at 11.3.0.46@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8805af62ec00, cur 1391078319 expire 1391078169 last 1391078090
Jan 30 10:38:39 sprig-oss2 kernel: : Lustre: Skipped 1 previous similar message