spirit-14.log:Apr 13 21:03:11 spirit-14 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101afd2800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-14.log:Apr 13 21:03:11 spirit-14 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101afd2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-14.log:Apr 13 21:03:11 spirit-14 kernel: LustreError: Skipped 12 previous similar messages spirit-14.log:Apr 13 21:03:11 spirit-14 kernel: LustreError: 12343:0:(file.c:176:ll_close_inode_openhandle()) lustre-clilmv-ffff88101afd2800: inode [0x200000409:0x70:0x0] mdc close failed: rc = -108 spirit-14.log:Apr 13 21:06:47 spirit-14 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101afd2800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-14.log:Apr 13 21:07:03 spirit-14 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101afd2800: operation obd_ping to node 192.168.1.4@o2ib failed: rc = -107 spirit-14.log:Apr 13 21:07:03 spirit-14 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101afd2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-14.log:Apr 13 21:07:03 spirit-14 kernel: LustreError: 13741:0:(vvp_io.c:1450:vvp_io_init()) lustre: refresh file layout [0x20000041a:0x1:0x0] error -5. spirit-14.log:Apr 13 21:07:03 spirit-14 kernel: LustreError: 13741:0:(vvp_io.c:1450:vvp_io_init()) Skipped 5 previous similar messages spirit-14.log:Apr 13 21:25:05 spirit-14 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101afd2800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-14.log:Apr 13 21:25:05 spirit-14 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101afd2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-14.log:Apr 13 21:25:05 spirit-14 kernel: LustreError: Skipped 12 previous similar messages spirit-15.log:Apr 13 21:03:11 spirit-15 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff880812595800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-15.log:Apr 13 21:03:11 spirit-15 kernel: LustreError: Skipped 6 previous similar messages spirit-15.log:Apr 13 21:03:11 spirit-15 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff880812595800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-15.log:Apr 13 21:03:11 spirit-15 kernel: LustreError: 12075:0:(file.c:176:ll_close_inode_openhandle()) lustre-clilmv-ffff880812595800: inode [0x200000406:0x82:0x0] mdc close failed: rc = -108 spirit-15.log:Apr 13 21:25:05 spirit-15 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff880812595800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-15.log:Apr 13 21:25:05 spirit-15 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff880812595800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-15.log:Apr 13 21:25:05 spirit-15 kernel: LustreError: Skipped 12 previous similar messages spirit-16.log:Apr 13 21:03:11 spirit-16 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101537b800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-16.log:Apr 13 21:03:11 spirit-16 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101537b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-16.log:Apr 13 21:03:11 spirit-16 kernel: LustreError: 12119:0:(file.c:176:ll_close_inode_openhandle()) lustre-clilmv-ffff88101537b800: inode [0x20000040a:0x71:0x0] mdc close failed: rc = -108 spirit-16.log:Apr 13 21:03:11 spirit-16 kernel: LustreError: Skipped 12 previous similar messages spirit-16.log:Apr 13 21:25:05 spirit-16 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101537b800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-16.log:Apr 13 21:25:05 spirit-16 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101537b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-16.log:Apr 13 21:25:05 spirit-16 kernel: LustreError: Skipped 9 previous similar messages spirit-17.log:Apr 13 21:03:11 spirit-17 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101b371000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-17.log:Apr 13 21:03:11 spirit-17 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101b371000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-17.log:Apr 13 21:03:11 spirit-17 kernel: LustreError: Skipped 8 previous similar messages spirit-17.log:Apr 13 21:25:05 spirit-17 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101b371000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-17.log:Apr 13 21:25:05 spirit-17 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101b371000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-17.log:Apr 13 21:25:05 spirit-17 kernel: LustreError: Skipped 12 previous similar messages spirit-18.log:Apr 13 21:03:11 spirit-18 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff880809e13000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-18.log:Apr 13 21:03:11 spirit-18 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff880809e13000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-18.log:Apr 13 21:03:11 spirit-18 kernel: LustreError: Skipped 12 previous similar messages spirit-18.log:Apr 13 21:25:05 spirit-18 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff880809e13000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-18.log:Apr 13 21:25:05 spirit-18 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff880809e13000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-18.log:Apr 13 21:25:05 spirit-18 kernel: LustreError: Skipped 12 previous similar messages spirit-19.log:Apr 13 21:03:11 spirit-19 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881019c1c000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-19.log:Apr 13 21:03:11 spirit-19 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881019c1c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-19.log:Apr 13 21:03:11 spirit-19 kernel: LustreError: Skipped 12 previous similar messages spirit-19.log:Apr 13 21:25:05 spirit-19 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881019c1c000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-19.log:Apr 13 21:25:05 spirit-19 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881019c1c000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-19.log:Apr 13 21:25:05 spirit-19 kernel: LustreError: Skipped 12 previous similar messages spirit-20.log:Apr 13 21:03:11 spirit-20 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881017f92000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-20.log:Apr 13 21:03:11 spirit-20 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881017f92000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-20.log:Apr 13 21:03:11 spirit-20 kernel: LustreError: Skipped 12 previous similar messages spirit-20.log:Apr 13 21:25:05 spirit-20 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881017f92000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-20.log:Apr 13 21:25:05 spirit-20 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881017f92000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-20.log:Apr 13 21:25:05 spirit-20 kernel: LustreError: Skipped 12 previous similar messages spirit-21.log:Apr 13 21:03:11 spirit-21 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101c85b800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-21.log:Apr 13 21:03:11 spirit-21 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101c85b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-21.log:Apr 13 21:03:11 spirit-21 kernel: LustreError: Skipped 12 previous similar messages spirit-21.log:Apr 13 21:25:05 spirit-21 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101c85b800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-21.log:Apr 13 21:25:05 spirit-21 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101c85b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-21.log:Apr 13 21:25:05 spirit-21 kernel: LustreError: Skipped 12 previous similar messages spirit-22.log:Apr 13 21:03:11 spirit-22 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101463e000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-22.log:Apr 13 21:03:11 spirit-22 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101463e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-22.log:Apr 13 21:03:11 spirit-22 kernel: LustreError: Skipped 12 previous similar messages spirit-22.log:Apr 13 21:25:05 spirit-22 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88101463e000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-22.log:Apr 13 21:25:05 spirit-22 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88101463e000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-22.log:Apr 13 21:25:05 spirit-22 kernel: LustreError: Skipped 12 previous similar messages spirit-23.log:Apr 13 21:03:11 spirit-23 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881018eeb000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-23.log:Apr 13 21:03:11 spirit-23 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881018eeb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-23.log:Apr 13 21:03:11 spirit-23 kernel: LustreError: Skipped 12 previous similar messages spirit-23.log:Apr 13 21:25:05 spirit-23 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881018eeb000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-23.log:Apr 13 21:25:05 spirit-23 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881018eeb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-23.log:Apr 13 21:25:05 spirit-23 kernel: LustreError: Skipped 12 previous similar messages spirit-24.log:Apr 13 21:03:11 spirit-24 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff8800b554f000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-24.log:Apr 13 21:03:11 spirit-24 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff8800b554f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-24.log:Apr 13 21:03:11 spirit-24 kernel: LustreError: Skipped 9 previous similar messages spirit-24.log:Apr 13 21:25:05 spirit-24 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff8800b554f000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-24.log:Apr 13 21:25:05 spirit-24 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff8800b554f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-24.log:Apr 13 21:25:05 spirit-24 kernel: LustreError: Skipped 12 previous similar messages spirit-25.log:Apr 13 21:03:11 spirit-25 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881015a9e800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-25.log:Apr 13 21:03:11 spirit-25 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881015a9e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-25.log:Apr 13 21:03:11 spirit-25 kernel: LustreError: Skipped 11 previous similar messages spirit-25.log:Apr 13 21:25:05 spirit-25 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881015a9e800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-25.log:Apr 13 21:25:05 spirit-25 kernel: LustreError: Skipped 6 previous similar messages spirit-25.log:Apr 13 21:25:05 spirit-25 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881015a9e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-26.log:Apr 13 21:03:11 spirit-26 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88100d81a800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-26.log:Apr 13 21:03:11 spirit-26 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88100d81a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-26.log:Apr 13 21:03:11 spirit-26 kernel: LustreError: Skipped 11 previous similar messages spirit-26.log:Apr 13 21:25:05 spirit-26 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88100d81a800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-26.log:Apr 13 21:25:05 spirit-26 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88100d81a800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-26.log:Apr 13 21:25:05 spirit-26 kernel: LustreError: Skipped 12 previous similar messages spirit-27.log:Apr 13 21:03:11 spirit-27 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881017f3f800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-27.log:Apr 13 21:03:11 spirit-27 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881017f3f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-27.log:Apr 13 21:03:11 spirit-27 kernel: LustreError: Skipped 12 previous similar messages spirit-27.log:Apr 13 21:25:05 spirit-27 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff881017f3f800: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-27.log:Apr 13 21:25:05 spirit-27 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff881017f3f800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-27.log:Apr 13 21:25:05 spirit-27 kernel: LustreError: Skipped 12 previous similar messages spirit-28.log:Apr 13 21:03:11 spirit-28 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88081138d000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-28.log:Apr 13 21:03:11 spirit-28 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88081138d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-28.log:Apr 13 21:03:11 spirit-28 kernel: LustreError: Skipped 12 previous similar messages spirit-28.log:Apr 13 21:25:05 spirit-28 kernel: LustreError: 11-0: lustre-MDT0000-mdc-ffff88081138d000: operation ldlm_enqueue to node 192.168.1.4@o2ib failed: rc = -116 spirit-28.log:Apr 13 21:25:05 spirit-28 kernel: LustreError: 167-0: lustre-MDT0000-mdc-ffff88081138d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. spirit-28.log:Apr 13 21:25:05 spirit-28 kernel: LustreError: Skipped 12 previous similar messages spirit-28.log:Apr 13 21:25:05 spirit-28 kernel: LustreError: 14737:0:(file.c:176:ll_close_inode_openhandle()) lustre-clilmv-ffff88081138d000: inode [0x200000419:0x6:0x0] mdc close failed: rc = -108 spirit-4.log:Apr 13 21:03:11 spirit-4 kernel: LustreError: 0:0:(ldlm_lockd.c:333:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 192.168.1.15@o2ib ns: mdt-lustre-MDT0000_UUID lock: ffff880818eb8200/0x22072ccf0ace0aa7 lrc: 4/0,0 mode: CR/CR res: [0x200000406:0x79:0x0].0x0 bits 0x8 rrc: 2 type: IBT flags: 0x60000400000020 nid: 192.168.1.15@o2ib remote: 0x24e8f0773c4a7b86 expref: 49 pid: 8233 timeout: 4306813538 lvb_type: 3 spirit-4.log:Apr 13 21:06:47 spirit-4 kernel: LustreError: 0:0:(ldlm_lockd.c:333:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 192.168.1.14@o2ib ns: mdt-lustre-MDT0000_UUID lock: ffff881024a08800/0x22072ccf0ace4a1e lrc: 4/0,0 mode: CR/CR res: [0x20000041a:0x1:0x0].0x0 bits 0x8 rrc: 18 type: IBT flags: 0x60000400000020 nid: 192.168.1.14@o2ib remote: 0x894cfc775d594d14 expref: 8 pid: 8248 timeout: 4307029582 lvb_type: 3 spirit-4.log:Apr 13 21:06:47 spirit-4 kernel: LustreError: 0:0:(ldlm_lockd.c:333:waiting_locks_callback()) Skipped 98 previous similar messages spirit-4.log:Apr 13 21:25:05 spirit-4 kernel: LustreError: 0:0:(ldlm_lockd.c:333:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 192.168.1.17@o2ib ns: mdt-lustre-MDT0000_UUID lock: ffff88102cbfae00/0x22072ccf0ace9390 lrc: 4/0,0 mode: CR/CR res: [0x200000415:0x3:0x0].0x0 bits 0x8 rrc: 2 type: IBT flags: 0x60000400000020 nid: 192.168.1.17@o2ib remote: 0xb7dfaac012988fa3 expref: 32 pid: 8315 timeout: 4308127658 lvb_type: 3